Jul 23 11:26:44.819428 ip-10-0-128-30 systemd[1]: Starting Kubernetes Kubelet... Jul 23 11:26:45.205206 ip-10-0-128-30 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:45.205206 ip-10-0-128-30 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 11:26:45.205206 ip-10-0-128-30 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:45.205206 ip-10-0-128-30 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 11:26:45.205206 ip-10-0-128-30 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:26:45.206189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.206084 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 11:26:45.211758 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211743 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211760 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211765 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211768 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211772 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211775 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211778 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211780 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211783 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211786 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211789 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211792 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211795 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211799 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211802 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211805 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.211802 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211808 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211811 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211814 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211816 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211819 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211823 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211825 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211828 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211830 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211833 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211835 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211838 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211840 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211843 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211845 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211848 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211850 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211853 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211855 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211858 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.212217 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211860 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211863 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211865 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211867 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211870 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211872 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211874 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211877 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211879 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211882 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211884 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211887 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211889 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211891 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211895 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211897 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211909 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211912 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211916 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211919 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.212693 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211921 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211924 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211927 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211929 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211932 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211934 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211938 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211941 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211944 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211946 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211949 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211951 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211954 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211956 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211959 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211961 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211964 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211967 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211969 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211973 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.213187 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211975 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211978 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211980 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211983 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211985 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211988 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211991 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211994 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.211997 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212000 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212003 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212006 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212009 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212011 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212014 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212017 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212019 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212022 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212024 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212026 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.213666 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212029 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212031 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212034 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212815 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212827 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212830 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212833 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212837 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212840 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212843 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212846 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212848 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212851 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212853 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212856 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212858 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212861 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212864 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212867 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212870 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.214170 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212873 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212875 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212878 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212880 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212883 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212885 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212888 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212890 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212892 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212895 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212898 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212900 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212903 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212905 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212908 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212910 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212914 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212916 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212919 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212921 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.214760 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212923 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212926 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212928 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212931 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212933 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212935 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212938 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212940 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212942 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212945 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212947 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212951 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212955 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212957 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212960 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212962 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212964 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212967 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212969 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.215519 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212972 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212975 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212978 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212980 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212982 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212985 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212988 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212990 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212992 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212996 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.212998 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213001 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213003 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213006 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213010 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213014 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213017 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213019 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213022 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.216042 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213024 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213027 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213029 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213032 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213035 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213037 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213040 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213042 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213045 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213047 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213050 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213052 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213054 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213058 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213062 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213065 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213067 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213070 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213073 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.216618 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213076 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213078 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213081 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213084 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213086 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213488 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213494 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213497 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213500 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213503 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213506 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213509 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213511 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213514 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213517 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213520 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213523 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213525 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213528 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213531 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.217088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213534 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213536 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213539 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213541 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213544 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213546 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213549 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213551 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213554 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213556 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213558 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213561 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213564 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213567 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213569 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213572 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213575 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213577 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213580 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213582 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.217611 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213585 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213587 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213590 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213592 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213594 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213597 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213599 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213602 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213604 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213606 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213609 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213612 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213615 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213618 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213621 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213623 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213625 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213628 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213630 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.218094 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213633 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213635 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213639 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213642 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213645 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213648 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213651 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213654 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213657 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213659 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213662 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213665 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213667 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213670 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213672 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213675 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213678 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213681 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213683 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213686 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.218573 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213688 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213691 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213693 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213695 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213698 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213701 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213703 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213706 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213709 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213713 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213716 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213718 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213720 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213723 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213725 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213728 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213732 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213735 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213738 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.219088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213741 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213744 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213746 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213749 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213751 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.213754 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213832 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213839 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213845 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213850 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213854 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213857 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213861 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213865 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213869 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213871 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213875 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213878 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213881 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213884 2564 flags.go:64] FLAG: --cgroup-root="" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213887 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213890 2564 flags.go:64] FLAG: --client-ca-file="" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213893 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 23 11:26:45.219584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213896 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213900 2564 flags.go:64] FLAG: --cluster-domain="" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213902 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213905 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213908 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213912 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213916 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213919 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213922 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213926 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213929 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213932 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213935 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213938 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213941 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213945 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213948 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213951 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213954 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213957 2564 flags.go:64] FLAG: --enable-server="true" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213959 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213964 2564 flags.go:64] FLAG: --event-burst="100" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213967 2564 flags.go:64] FLAG: --event-qps="50" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213970 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 11:26:45.220201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213973 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213976 2564 flags.go:64] FLAG: --eviction-hard="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213979 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213982 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213985 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213988 2564 flags.go:64] FLAG: --eviction-soft="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213991 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213994 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.213997 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214000 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214002 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214005 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214008 2564 flags.go:64] FLAG: --feature-gates="" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214012 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214018 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214021 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214024 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214027 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214030 2564 flags.go:64] FLAG: --help="false" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214033 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214036 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214039 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214041 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214045 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 11:26:45.220775 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214048 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214051 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214054 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214056 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214059 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214062 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214065 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214068 2564 flags.go:64] FLAG: --kube-reserved="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214071 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214073 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214078 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214081 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214083 2564 flags.go:64] FLAG: --lock-file="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214087 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214090 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214094 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214099 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214101 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214104 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214107 2564 flags.go:64] FLAG: --logging-format="text" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214110 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214113 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214117 2564 flags.go:64] FLAG: --manifest-url="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214120 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214124 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 11:26:45.221478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214127 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214131 2564 flags.go:64] FLAG: --max-pods="110" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214134 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214137 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214140 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214142 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214159 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214162 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214165 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214172 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214175 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214178 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214181 2564 flags.go:64] FLAG: --pod-cidr="" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214184 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214189 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214192 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214195 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214199 2564 flags.go:64] FLAG: --port="10250" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214202 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214205 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02cfb29293e8231a1" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214208 2564 flags.go:64] FLAG: --qos-reserved="" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214211 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214213 2564 flags.go:64] FLAG: --register-node="true" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214217 2564 flags.go:64] FLAG: --register-with-taints="" Jul 23 11:26:45.222093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214220 2564 flags.go:64] FLAG: --registry-burst="10" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214223 2564 flags.go:64] FLAG: --registry-qps="5" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214226 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214229 2564 flags.go:64] FLAG: --reserved-memory="" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214233 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214236 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214240 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214243 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214246 2564 flags.go:64] FLAG: --runonce="false" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214248 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214251 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214254 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214257 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214260 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214263 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214267 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214270 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214272 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214276 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214278 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214281 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214284 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214286 2564 flags.go:64] FLAG: --system-cgroups="" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214289 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214294 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 11:26:45.222743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214298 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214301 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214305 2564 flags.go:64] FLAG: --tls-min-version="" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214307 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214310 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214313 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214316 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214319 2564 flags.go:64] FLAG: --v="2" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214323 2564 flags.go:64] FLAG: --version="false" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214327 2564 flags.go:64] FLAG: --vmodule="" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214330 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.214333 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214579 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214598 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214603 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214608 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214757 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214765 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214770 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214775 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214780 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214785 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.223368 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214794 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214802 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214808 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214813 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214819 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214824 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214835 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214840 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214845 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214849 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214854 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214859 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214863 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214868 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214872 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214878 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214883 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214887 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214896 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.223939 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214903 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214909 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214915 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214920 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214925 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214929 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214934 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214940 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214946 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214951 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214955 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214960 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214970 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214976 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214980 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214985 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214989 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214994 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.214999 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215003 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.224437 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215008 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215012 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215018 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215022 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215032 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215036 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215040 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215045 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215049 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215054 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215059 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215063 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215068 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215072 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215076 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215080 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215085 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215094 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215099 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215106 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.224923 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215111 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215116 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215120 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215125 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215132 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215138 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215143 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215166 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215172 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215176 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215186 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215191 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215196 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215201 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215206 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215210 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215215 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215219 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215223 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215227 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.225422 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215232 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215236 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215246 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215251 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215255 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215259 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215264 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215268 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215273 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.215277 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.215945 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.223879 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.223896 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.223923 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.223929 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 11:26:45.225894 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.223984 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.223989 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.223992 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.223995 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.223999 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224002 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224005 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224008 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224011 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224013 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224016 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224019 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224021 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224024 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224027 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224029 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224033 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224037 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224040 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224043 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.226291 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224045 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224048 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224051 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224054 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224057 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224059 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224062 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224065 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224068 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224071 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224075 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224079 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224082 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224085 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224088 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224090 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224093 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224095 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224099 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.226778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224102 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224105 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224108 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224111 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224113 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224116 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224119 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224122 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224124 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224127 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224129 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224132 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224135 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224138 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224140 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224143 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224161 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224165 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224168 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224171 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.227307 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224173 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224176 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224180 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224183 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224185 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224188 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224191 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224193 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224195 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224198 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224201 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224203 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224206 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224208 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224211 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224213 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224216 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224219 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224222 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224224 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.227786 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224227 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224229 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224232 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224234 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224237 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224239 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224242 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224244 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224247 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224249 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224251 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224254 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224256 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224259 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224262 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224265 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224268 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224270 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224272 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.228458 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224275 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.224280 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224410 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224414 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224419 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224422 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224425 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224428 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224432 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224434 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224437 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224439 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224443 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224446 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224448 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:26:45.228944 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224451 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224454 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224456 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224459 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224461 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224465 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224468 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224471 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224473 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224476 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224478 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224481 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224484 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224486 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224489 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224492 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224494 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224496 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224499 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224501 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:26:45.229574 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224504 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224506 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224509 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224511 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224513 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224516 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224519 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224522 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224525 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224527 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224530 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224533 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224535 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224537 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224540 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224542 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224544 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224547 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224549 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:26:45.230313 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224552 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224554 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224556 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224558 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224561 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224563 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224566 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224569 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224571 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224574 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224576 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224578 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224581 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224583 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224585 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224588 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224590 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224593 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224595 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224598 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:26:45.230867 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224600 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224602 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224605 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224607 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224610 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224612 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224614 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224617 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224619 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224622 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224624 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224627 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224629 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224632 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224634 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224636 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224639 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224641 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224644 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224646 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:26:45.231536 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224649 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224651 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224653 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224656 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224658 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224661 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:45.224663 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.224667 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.224783 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.226687 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.227965 2564 server.go:1019] "Starting client certificate rotation" Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.228086 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.228127 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:26:45.232060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.230239 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:26:45.232458 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.232114 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:26:45.247001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.246981 2564 log.go:25] "Validated CRI v1 runtime API" Jul 23 11:26:45.250660 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.250641 2564 log.go:25] "Validated CRI v1 image API" Jul 23 11:26:45.251787 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.251757 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 11:26:45.254113 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.254095 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 88cecebd-0d52-4c4d-9ef1-051af7d7f425:/dev/nvme0n1p3 e31e8162-e1e6-44a0-b336-df9922ed1c99:/dev/nvme0n1p4] Jul 23 11:26:45.254195 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.254113 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 11:26:45.256278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.256258 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:26:45.259122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.259015 2564 manager.go:216] Machine: {Timestamp:2026-07-23 11:26:45.257679517 +0000 UTC m=+0.327290536 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099457 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2acd034c13a360dfa540cf47657ae0 SystemUUID:ec2acd03-4c13-a360-dfa5-40cf47657ae0 BootID:592990c4-332c-46b2-9d7b-bd421af056c6 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:1f:bc:51:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:1f:bc:51:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:da:25:b4:0b:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 11:26:45.259122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.259118 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 11:26:45.259264 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.259208 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 11:26:45.262097 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.262075 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 11:26:45.262237 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.262100 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 11:26:45.262774 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.262764 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 11:26:45.262809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.262776 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 11:26:45.262809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.262785 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:26:45.263459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.263449 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:26:45.264546 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.264537 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 11:26:45.267384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267373 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:26:45.267502 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267492 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 11:26:45.267541 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267532 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 23 11:26:45.267570 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267551 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 11:26:45.267570 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267562 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 11:26:45.267627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267572 2564 kubelet.go:398] "Adding apiserver pod source" Jul 23 11:26:45.267627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267581 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 11:26:45.267947 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267925 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:26:45.267988 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.267954 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:26:45.270378 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.270358 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 11:26:45.272094 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.272073 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 11:26:45.272162 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.272104 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 11:26:45.273408 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273395 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273412 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273419 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273424 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273432 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273439 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273447 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 11:26:45.273473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273453 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273478 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273485 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273498 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273503 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273527 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 11:26:45.273670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.273532 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 11:26:45.276761 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.276745 2564 server.go:1268] "Started kubelet" Jul 23 11:26:45.276854 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.276833 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 11:26:45.276954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.276900 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 11:26:45.277001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.276978 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 11:26:45.278296 ip-10-0-128-30 systemd[1]: Started Kubernetes Kubelet. Jul 23 11:26:45.278529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.278508 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 11:26:45.278602 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.278507 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 11:26:45.278708 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.278673 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 11:26:45.278959 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.278839 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 11:26:45.279308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.279252 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dsw2t" Jul 23 11:26:45.280202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.280184 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 23 11:26:45.283686 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.283663 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 11:26:45.284758 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.284738 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 11:26:45.284840 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.284829 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.284895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.284739 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 11:26:45.285297 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285281 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 11:26:45.285402 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285363 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 11:26:45.285500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285485 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 11:26:45.285579 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285565 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 23 11:26:45.285579 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285578 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 23 11:26:45.285689 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285592 2564 factory.go:55] Registering systemd factory Jul 23 11:26:45.285689 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285604 2564 factory.go:223] Registration of the systemd container factory successfully Jul 23 11:26:45.285913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285898 2564 factory.go:153] Registering CRI-O factory Jul 23 11:26:45.285972 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285919 2564 factory.go:223] Registration of the crio container factory successfully Jul 23 11:26:45.286047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285972 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 11:26:45.286047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.285997 2564 factory.go:103] Registering Raw factory Jul 23 11:26:45.286047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.286013 2564 manager.go:1196] Started watching for new ooms in manager Jul 23 11:26:45.286628 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.286615 2564 manager.go:349] Starting recovery of all containers Jul 23 11:26:45.286840 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.285805 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-30.ec2.internal.18c4e79b0180ae99 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-30.ec2.internal,UID:ip-10-0-128-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-30.ec2.internal,},FirstTimestamp:2026-07-23 11:26:45.276716697 +0000 UTC m=+0.346327715,LastTimestamp:2026-07-23 11:26:45.276716697 +0000 UTC m=+0.346327715,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-30.ec2.internal,}" Jul 23 11:26:45.288999 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.288975 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dsw2t" Jul 23 11:26:45.292633 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.292603 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 11:26:45.292748 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.292709 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 11:26:45.296529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.296423 2564 manager.go:354] Recovery completed Jul 23 11:26:45.300768 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.300756 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.309876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.309861 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.309949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.309890 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.309949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.309900 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.310380 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.310366 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 11:26:45.310380 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.310378 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 11:26:45.310477 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.310393 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:26:45.310477 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.310424 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.311564 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311551 2564 policy_none.go:49] "None policy: Start" Jul 23 11:26:45.311621 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311567 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 11:26:45.311621 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311577 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 11:26:45.311621 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311603 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.311722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311629 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.311722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.311639 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.312626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.312614 2564 policy_none.go:47] "Start" Jul 23 11:26:45.349764 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.349743 2564 manager.go:337] "Starting Device Plugin manager" Jul 23 11:26:45.349856 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.349777 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 11:26:45.349856 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.349786 2564 server.go:87] "Starting device plugin registration server" Jul 23 11:26:45.350036 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.350023 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 11:26:45.350087 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.350039 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 11:26:45.350138 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.350121 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 11:26:45.350239 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.350226 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 11:26:45.350291 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.350241 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 11:26:45.350766 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.350750 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 11:26:45.350827 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.350791 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.447178 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.447124 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 11:26:45.448562 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.448543 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 11:26:45.448629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.448594 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 11:26:45.448629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.448625 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 11:26:45.448705 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.448668 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 11:26:45.450136 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.450114 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.451097 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.451080 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.451182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.451112 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.451182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.451127 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.451182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.451175 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.451728 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.451708 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:45.459515 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.459466 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.459515 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.459487 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-30.ec2.internal\": node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.482422 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.482395 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.548901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.548859 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal"] Jul 23 11:26:45.549084 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.548946 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.551085 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.551063 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.551214 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.551099 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.551214 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.551114 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.552526 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.552512 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.552683 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.552667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.552744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.552708 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.554496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554479 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.554496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554492 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.554647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554510 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.554647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554513 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.554647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554524 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.554647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.554530 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.555779 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.555762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.555867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.555790 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:26:45.557834 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.557817 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:26:45.557934 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.557847 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:26:45.557934 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.557863 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:26:45.582820 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.582792 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.587094 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.587073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.587216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.587100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.587216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.587120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e524dd64ccddd7fadb096176a3752af3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-30.ec2.internal\" (UID: \"e524dd64ccddd7fadb096176a3752af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.588926 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.588910 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-30.ec2.internal\" not found" node="ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.593545 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.593529 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-30.ec2.internal\" not found" node="ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.683555 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.683518 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.687898 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.687981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.687981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e524dd64ccddd7fadb096176a3752af3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-30.ec2.internal\" (UID: \"e524dd64ccddd7fadb096176a3752af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.687981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e524dd64ccddd7fadb096176a3752af3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-30.ec2.internal\" (UID: \"e524dd64ccddd7fadb096176a3752af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.688120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.688120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.687988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4e9d4dfd88fb1f06ac352c38a2cfb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal\" (UID: \"6f4e9d4dfd88fb1f06ac352c38a2cfb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.784426 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.784343 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.884796 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.884756 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:45.892158 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.892119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.895960 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:45.895879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:45.985550 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:45.985509 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.173637 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.173538 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.273983 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.273941 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.315244 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.315220 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 11:26:47.315804 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.315366 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:26:47.349188 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.349155 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:47.369712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.369689 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:47.374034 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.374012 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.381294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.381265 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:21:46 +0000 UTC" deadline="2028-02-22 19:39:59.982632901 +0000 UTC" Jul 23 11:26:47.381369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.381294 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13904h13m12.601342729s" Jul 23 11:26:47.474712 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.474681 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.575240 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.575206 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.601969 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:47.601938 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e9d4dfd88fb1f06ac352c38a2cfb2.slice/crio-f79915cbe4ca98d11707fc20120f61e068b7a6c6ca84acf434c2225770a34136 WatchSource:0}: Error finding container f79915cbe4ca98d11707fc20120f61e068b7a6c6ca84acf434c2225770a34136: Status 404 returned error can't find the container with id f79915cbe4ca98d11707fc20120f61e068b7a6c6ca84acf434c2225770a34136 Jul 23 11:26:47.602177 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:47.602166 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode524dd64ccddd7fadb096176a3752af3.slice/crio-180940a3ed91d651eca61776dce99da92725964ea8b87350c59f1a0602df9fc5 WatchSource:0}: Error finding container 180940a3ed91d651eca61776dce99da92725964ea8b87350c59f1a0602df9fc5: Status 404 returned error can't find the container with id 180940a3ed91d651eca61776dce99da92725964ea8b87350c59f1a0602df9fc5 Jul 23 11:26:47.606267 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.606250 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:26:47.675854 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.675815 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.776355 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:47.776295 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-30.ec2.internal\" not found" Jul 23 11:26:47.856389 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.856365 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:47.873229 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.873204 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" Jul 23 11:26:47.884361 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.884336 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:26:47.885492 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.885478 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" Jul 23 11:26:47.893706 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:47.893682 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:26:48.356679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.356648 2564 apiserver.go:52] "Watching apiserver" Jul 23 11:26:48.370185 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.370150 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 11:26:48.371246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.371216 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hfvsb","kube-system/konnectivity-agent-96klp","kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal","openshift-multus/multus-additional-cni-plugins-zv2tw","openshift-multus/multus-l6glc","openshift-network-operator/iptables-alerter-85jmq","openshift-cluster-node-tuning-operator/tuned-blnrj","openshift-dns/node-resolver-6xkh9","openshift-image-registry/node-ca-bs9l8","openshift-multus/network-metrics-daemon-qqn65","openshift-network-diagnostics/network-check-target-2dfxr","openshift-ovn-kubernetes/ovnkube-node-4kjj5"] Jul 23 11:26:48.373200 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.373178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.373602 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.373567 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 11:26:48.376197 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.375426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.376368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.376309 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 11:26:48.376802 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.376705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 11:26:48.376915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.376861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kl9lq\"" Jul 23 11:26:48.377180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.377090 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.377305 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.377237 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.377305 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.377300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.379121 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.378813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.379825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380811 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380913 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q7886\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.380993 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381058 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381098 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381187 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9pd9p\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381235 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkvkg\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.381609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.382432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.381838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.382432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.382156 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.382432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.382220 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 11:26:48.382432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.382404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7knnw\"" Jul 23 11:26:48.382922 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.382902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-k8s-cni-cncf-io\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-hostroot\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzwr\" (UniqueName: \"kubernetes.io/projected/d2044830-359c-4bb5-b06d-7080785eaa70-kube-api-access-nvzwr\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-system-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-multus\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-multus-daemon-config\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-multus-certs\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.383530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-cnibin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-netns\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-serviceca\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dhk\" (UniqueName: \"kubernetes.io/projected/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-kube-api-access-f4dhk\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-socket-dir-parent\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-bin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-kubelet\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-conf-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-etc-kubernetes\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-host\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-os-release\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.383935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-cni-binary-copy\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.384821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.384190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:48.384821 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.384250 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:48.386008 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.385812 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vk9q9\"" Jul 23 11:26:48.386008 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.385823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 11:26:48.386153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.386053 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.386153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.386110 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 11:26:48.386341 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.386320 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d2gdp\"" Jul 23 11:26:48.386915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.386887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.387694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.387674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.388196 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.388175 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:26:48.389340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.389322 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.389977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.389959 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 11:26:48.390095 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.390081 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r6bn9\"" Jul 23 11:26:48.390385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.390368 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 11:26:48.391363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.391347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.391460 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.391414 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:48.391841 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.391813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 11:26:48.392036 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.392021 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9g76b\"" Jul 23 11:26:48.392369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.392354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 11:26:48.392737 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.392719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.392846 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.392826 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:48.406146 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.406120 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vw8hb" Jul 23 11:26:48.414140 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.414116 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vw8hb" Jul 23 11:26:48.420844 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.420824 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:26:48.475130 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.475096 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 11:26:48.484255 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qmz\" (UniqueName: \"kubernetes.io/projected/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-kube-api-access-r4qmz\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-etc-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-dbus\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr884\" (UniqueName: \"kubernetes.io/projected/bd6840b5-931b-4a08-8db0-f068a43d2a8d-kube-api-access-cr884\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-host\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-modprobe-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.484422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbrz\" (UniqueName: \"kubernetes.io/projected/30f59c66-9b75-4cc3-aea4-020344e4d396-kube-api-access-zrbrz\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4c7\" (UniqueName: \"kubernetes.io/projected/89e83780-8bf1-4ece-b2da-a4024c2576b2-kube-api-access-rl4c7\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-kubelet\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-node-log\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-system-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-sys\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrs2\" (UniqueName: \"kubernetes.io/projected/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-kube-api-access-bmrs2\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484597 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-host\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-conf\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7655ae45-899d-45ed-a10f-6286660f2c51-hosts-file\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.484749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-bin\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-system-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-socket-dir-parent\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-kubelet\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.484980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-kubelet\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-socket-dir-parent\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01887123-dc9b-428d-967f-8d195ad04afe-konnectivity-ca\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-config\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-ovn\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-log-socket\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-netd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-os-release\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.485327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-cni-binary-copy\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-os-release\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzwr\" (UniqueName: \"kubernetes.io/projected/d2044830-359c-4bb5-b06d-7080785eaa70-kube-api-access-nvzwr\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-systemd\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-netns\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-env-overrides\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovn-node-metrics-cert\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-sys-fs\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-netns\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f59c66-9b75-4cc3-aea4-020344e4d396-iptables-alerter-script\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f59c66-9b75-4cc3-aea4-020344e4d396-host-slash\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-cni-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-netns\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-conf-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-cni-binary-copy\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-multus-conf-dir\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.485968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01887123-dc9b-428d-967f-8d195ad04afe-agent-certs\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52bg\" (UniqueName: \"kubernetes.io/projected/7655ae45-899d-45ed-a10f-6286660f2c51-kube-api-access-j52bg\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-cnibin\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-multus\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-multus\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-var-lib-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kkx\" (UniqueName: \"kubernetes.io/projected/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kube-api-access-d9kkx\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dhk\" (UniqueName: \"kubernetes.io/projected/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-kube-api-access-f4dhk\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-lib-modules\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-var-lib-kubelet\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-systemd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-system-cni-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-os-release\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.486667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-slash\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-bin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-etc-kubernetes\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysconfig\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-var-lib-cni-bin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-run\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-tuned\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-etc-kubernetes\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655ae45-899d-45ed-a10f-6286660f2c51-tmp-dir\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-systemd-units\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-k8s-cni-cncf-io\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-hostroot\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-kubernetes\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486804 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-k8s-cni-cncf-io\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-script-lib\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-socket-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.487481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-hostroot\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-registration-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-device-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-multus-daemon-config\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-multus-certs\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-host\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-kubelet-config\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.486992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-cnibin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-serviceca\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-tmp\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-host-run-multus-certs\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2044830-359c-4bb5-b06d-7080785eaa70-cnibin\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2044830-359c-4bb5-b06d-7080785eaa70-multus-daemon-config\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.488293 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.487473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-serviceca\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.492026 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.492005 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 11:26:48.496122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.496098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzwr\" (UniqueName: \"kubernetes.io/projected/d2044830-359c-4bb5-b06d-7080785eaa70-kube-api-access-nvzwr\") pod \"multus-l6glc\" (UID: \"d2044830-359c-4bb5-b06d-7080785eaa70\") " pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.497621 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.497580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dhk\" (UniqueName: \"kubernetes.io/projected/04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16-kube-api-access-f4dhk\") pod \"node-ca-bs9l8\" (UID: \"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16\") " pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.542530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.542475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" event={"ID":"6f4e9d4dfd88fb1f06ac352c38a2cfb2","Type":"ContainerStarted","Data":"f79915cbe4ca98d11707fc20120f61e068b7a6c6ca84acf434c2225770a34136"} Jul 23 11:26:48.544200 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.544171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" event={"ID":"e524dd64ccddd7fadb096176a3752af3","Type":"ContainerStarted","Data":"180940a3ed91d651eca61776dce99da92725964ea8b87350c59f1a0602df9fc5"} Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-host\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-kubelet-config\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-tmp\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qmz\" (UniqueName: \"kubernetes.io/projected/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-kube-api-access-r4qmz\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-host\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-etc-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-dbus\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-etc-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr884\" (UniqueName: \"kubernetes.io/projected/bd6840b5-931b-4a08-8db0-f068a43d2a8d-kube-api-access-cr884\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-kubelet-config\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-modprobe-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbrz\" (UniqueName: \"kubernetes.io/projected/30f59c66-9b75-4cc3-aea4-020344e4d396-kube-api-access-zrbrz\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4c7\" (UniqueName: \"kubernetes.io/projected/89e83780-8bf1-4ece-b2da-a4024c2576b2-kube-api-access-rl4c7\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-kubelet\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-node-log\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.591479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-sys\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrs2\" (UniqueName: \"kubernetes.io/projected/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-kube-api-access-bmrs2\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab9726ae-8666-45a9-89d2-91e32363b1ee-dbus\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-kubelet\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-conf\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.588938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7655ae45-899d-45ed-a10f-6286660f2c51-hosts-file\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.589163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-sys\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-bin\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01887123-dc9b-428d-967f-8d195ad04afe-konnectivity-ca\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-config\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.592548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.591844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-modprobe-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-tmp\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-ovn\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-log-socket\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-netd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.596837 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-systemd\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-netns\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-env-overrides\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovn-node-metrics-cert\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-sys-fs\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.596969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f59c66-9b75-4cc3-aea4-020344e4d396-iptables-alerter-script\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f59c66-9b75-4cc3-aea4-020344e4d396-host-slash\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.597278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01887123-dc9b-428d-967f-8d195ad04afe-agent-certs\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52bg\" (UniqueName: \"kubernetes.io/projected/7655ae45-899d-45ed-a10f-6286660f2c51-kube-api-access-j52bg\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-cnibin\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-var-lib-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kkx\" (UniqueName: \"kubernetes.io/projected/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kube-api-access-d9kkx\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-lib-modules\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-var-lib-kubelet\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-systemd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.597782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-system-cni-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-os-release\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-slash\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysconfig\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-run\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.597987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-tuned\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655ae45-899d-45ed-a10f-6286660f2c51-tmp-dir\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.598245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.598551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-systemd-units\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.598551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-bin\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.598647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-log-socket\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.598647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-systemd-units\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598748 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-systemd\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-kubernetes\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-ovn\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.598653 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-cni-netd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-run-netns\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.598972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-node-log\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599102 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-conf\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.599273 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:26:49.099103239 +0000 UTC m=+3.081100720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7655ae45-899d-45ed-a10f-6286660f2c51-hosts-file\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-script-lib\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-socket-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-registration-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-device-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.602337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-device-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.599809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01887123-dc9b-428d-967f-8d195ad04afe-konnectivity-ca\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-env-overrides\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600253 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600274 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-cnibin\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600286 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600360 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:49.10034323 +0000 UTC m=+3.082340726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89e83780-8bf1-4ece-b2da-a4024c2576b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-script-lib\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysctl-d\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-registration-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f59c66-9b75-4cc3-aea4-020344e4d396-host-slash\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-var-lib-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600532 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-slash\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603263 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:48.600586 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:26:49.100570963 +0000 UTC m=+3.082568451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-socket-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.600751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-sys-fs\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovnkube-config\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-openvswitch\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/30f59c66-9b75-4cc3-aea4-020344e4d396-iptables-alerter-script\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-var-lib-kubelet\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-run-systemd\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7655ae45-899d-45ed-a10f-6286660f2c51-tmp-dir\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-sysconfig\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-lib-modules\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.601731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-run\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.602317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-os-release\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.602685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-kubernetes\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.602777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-system-cni-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.603991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.603025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89e83780-8bf1-4ece-b2da-a4024c2576b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.604800 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.603065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.604800 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.604614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-ovn-node-metrics-cert\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.605593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.605544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4c7\" (UniqueName: \"kubernetes.io/projected/89e83780-8bf1-4ece-b2da-a4024c2576b2-kube-api-access-rl4c7\") pod \"multus-additional-cni-plugins-zv2tw\" (UID: \"89e83780-8bf1-4ece-b2da-a4024c2576b2\") " pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.605991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.605857 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr884\" (UniqueName: \"kubernetes.io/projected/bd6840b5-931b-4a08-8db0-f068a43d2a8d-kube-api-access-cr884\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:48.605991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.605884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qmz\" (UniqueName: \"kubernetes.io/projected/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-kube-api-access-r4qmz\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.607057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.606967 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61-etc-tuned\") pod \"tuned-blnrj\" (UID: \"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61\") " pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.607829 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.607799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01887123-dc9b-428d-967f-8d195ad04afe-agent-certs\") pod \"konnectivity-agent-96klp\" (UID: \"01887123-dc9b-428d-967f-8d195ad04afe\") " pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.608176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.608153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrs2\" (UniqueName: \"kubernetes.io/projected/7ecbc862-e2ba-430c-9bab-dfcb931bc27b-kube-api-access-bmrs2\") pod \"ovnkube-node-4kjj5\" (UID: \"7ecbc862-e2ba-430c-9bab-dfcb931bc27b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.614414 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.614392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52bg\" (UniqueName: \"kubernetes.io/projected/7655ae45-899d-45ed-a10f-6286660f2c51-kube-api-access-j52bg\") pod \"node-resolver-6xkh9\" (UID: \"7655ae45-899d-45ed-a10f-6286660f2c51\") " pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:48.614926 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.614908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbrz\" (UniqueName: \"kubernetes.io/projected/30f59c66-9b75-4cc3-aea4-020344e4d396-kube-api-access-zrbrz\") pod \"iptables-alerter-85jmq\" (UID: \"30f59c66-9b75-4cc3-aea4-020344e4d396\") " pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.615023 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.614953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kkx\" (UniqueName: \"kubernetes.io/projected/af9dbea7-fb82-4488-aa5e-3b1aa99ddf45-kube-api-access-d9kkx\") pod \"aws-ebs-csi-driver-node-p6n5l\" (UID: \"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.689300 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.689263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6glc" Jul 23 11:26:48.705500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.705470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs9l8" Jul 23 11:26:48.714334 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.714307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:26:48.722269 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.722068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" Jul 23 11:26:48.732135 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.732112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-85jmq" Jul 23 11:26:48.739702 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.739683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" Jul 23 11:26:48.747314 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.747292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blnrj" Jul 23 11:26:48.754959 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.754941 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:26:48.761433 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:48.761414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6xkh9" Jul 23 11:26:49.103373 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.103339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:49.103560 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.103384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:49.103560 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.103410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:49.103560 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103527 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103593 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.103578795 +0000 UTC m=+4.085576276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103528 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103671 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.103658958 +0000 UTC m=+4.085656446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103529 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103699 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:49.103735 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103711 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:49.104074 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.103747 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:50.103738582 +0000 UTC m=+4.085736094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:49.416553 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.416473 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:21:48 +0000 UTC" deadline="2027-12-24 18:00:21.461639944 +0000 UTC" Jul 23 11:26:49.416553 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.416502 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12462h33m32.045141243s" Jul 23 11:26:49.537508 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.537471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:49.537671 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:49.537517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:49.537671 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.537598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:49.537807 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:49.537687 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:49.592876 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.592848 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9dbea7_fb82_4488_aa5e_3b1aa99ddf45.slice/crio-bf35c2b5479e129ca43a57a061eb2444c2bdce1893047f6897c482a10da5b45c WatchSource:0}: Error finding container bf35c2b5479e129ca43a57a061eb2444c2bdce1893047f6897c482a10da5b45c: Status 404 returned error can't find the container with id bf35c2b5479e129ca43a57a061eb2444c2bdce1893047f6897c482a10da5b45c Jul 23 11:26:49.611095 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.611071 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b1670e_e6b4_4904_ad2e_a8ed9e7a8d61.slice/crio-5e039dfca040addb595c57b4dc83ef2bc17de47ae6c422f006eb7fbe72e27918 WatchSource:0}: Error finding container 5e039dfca040addb595c57b4dc83ef2bc17de47ae6c422f006eb7fbe72e27918: Status 404 returned error can't find the container with id 5e039dfca040addb595c57b4dc83ef2bc17de47ae6c422f006eb7fbe72e27918 Jul 23 11:26:49.611669 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.611447 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cb6eb4_10c9_4b3c_9aaf_4c6ce51ccc16.slice/crio-2ba05e179c6e02f6848f436fb1c5a2c6506398a38cd5201d001a484627ebc583 WatchSource:0}: Error finding container 2ba05e179c6e02f6848f436fb1c5a2c6506398a38cd5201d001a484627ebc583: Status 404 returned error can't find the container with id 2ba05e179c6e02f6848f436fb1c5a2c6506398a38cd5201d001a484627ebc583 Jul 23 11:26:49.612367 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.612345 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2044830_359c_4bb5_b06d_7080785eaa70.slice/crio-28cef93b1819c620eafc321623cbe993577eb8dcbbfef7bdd480e7def6c15e17 WatchSource:0}: Error finding container 28cef93b1819c620eafc321623cbe993577eb8dcbbfef7bdd480e7def6c15e17: Status 404 returned error can't find the container with id 28cef93b1819c620eafc321623cbe993577eb8dcbbfef7bdd480e7def6c15e17 Jul 23 11:26:49.613960 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.613559 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7655ae45_899d_45ed_a10f_6286660f2c51.slice/crio-55616515a91e89ff70791ed3d93889b7b2cc052d3e0ac7007f20fa6be72a659f WatchSource:0}: Error finding container 55616515a91e89ff70791ed3d93889b7b2cc052d3e0ac7007f20fa6be72a659f: Status 404 returned error can't find the container with id 55616515a91e89ff70791ed3d93889b7b2cc052d3e0ac7007f20fa6be72a659f Jul 23 11:26:49.618330 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.617540 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f59c66_9b75_4cc3_aea4_020344e4d396.slice/crio-02e8c1b14aa4ce46dbbdf843a15fb681cf15083a205b526822e61906a6e686ba WatchSource:0}: Error finding container 02e8c1b14aa4ce46dbbdf843a15fb681cf15083a205b526822e61906a6e686ba: Status 404 returned error can't find the container with id 02e8c1b14aa4ce46dbbdf843a15fb681cf15083a205b526822e61906a6e686ba Jul 23 11:26:49.620471 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:26:49.620077 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e83780_8bf1_4ece_b2da_a4024c2576b2.slice/crio-686fdf2aef87c924fb4a1df47556f8e99df3d39f91d035968157b980c94e3dc5 WatchSource:0}: Error finding container 686fdf2aef87c924fb4a1df47556f8e99df3d39f91d035968157b980c94e3dc5: Status 404 returned error can't find the container with id 686fdf2aef87c924fb4a1df47556f8e99df3d39f91d035968157b980c94e3dc5 Jul 23 11:26:50.112001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.111961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:50.112208 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.112027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:50.112208 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.112090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:50.112208 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112140 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112215 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:26:52.11219559 +0000 UTC m=+6.094193072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112227 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112247 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112260 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112312 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:52.112296641 +0000 UTC m=+6.094294140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:50.112396 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112372 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:50.112661 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.112402 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:26:52.112391898 +0000 UTC m=+6.094389393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:50.417383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.417297 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:21:48 +0000 UTC" deadline="2028-05-03 21:23:31.190981488 +0000 UTC" Jul 23 11:26:50.417383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.417339 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15609h56m40.773647264s" Jul 23 11:26:50.537936 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.537889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:50.538123 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:50.538036 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:50.571874 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.570892 2564 generic.go:356] "Generic (PLEG): container finished" podID="6f4e9d4dfd88fb1f06ac352c38a2cfb2" containerID="7014de633297ab71a76d8f960b72ed6150dc20e8b42a41bb54d42819c40c23a2" exitCode=0 Jul 23 11:26:50.571874 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.571001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" event={"ID":"6f4e9d4dfd88fb1f06ac352c38a2cfb2","Type":"ContainerDied","Data":"7014de633297ab71a76d8f960b72ed6150dc20e8b42a41bb54d42819c40c23a2"} Jul 23 11:26:50.586395 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.586303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" event={"ID":"e524dd64ccddd7fadb096176a3752af3","Type":"ContainerStarted","Data":"a54c8f992b6507271b1b2ac3f5a343777a95908d1b7e9520bf4f757f873d5fd1"} Jul 23 11:26:50.600411 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.600295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerStarted","Data":"686fdf2aef87c924fb4a1df47556f8e99df3d39f91d035968157b980c94e3dc5"} Jul 23 11:26:50.637166 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.637073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-85jmq" event={"ID":"30f59c66-9b75-4cc3-aea4-020344e4d396","Type":"ContainerStarted","Data":"02e8c1b14aa4ce46dbbdf843a15fb681cf15083a205b526822e61906a6e686ba"} Jul 23 11:26:50.646809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.646750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6xkh9" event={"ID":"7655ae45-899d-45ed-a10f-6286660f2c51","Type":"ContainerStarted","Data":"55616515a91e89ff70791ed3d93889b7b2cc052d3e0ac7007f20fa6be72a659f"} Jul 23 11:26:50.663427 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.663386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs9l8" event={"ID":"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16","Type":"ContainerStarted","Data":"2ba05e179c6e02f6848f436fb1c5a2c6506398a38cd5201d001a484627ebc583"} Jul 23 11:26:50.694077 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.693912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96klp" event={"ID":"01887123-dc9b-428d-967f-8d195ad04afe","Type":"ContainerStarted","Data":"0966fc32aa95c2d94087aa2316784df88441f30757fb2139f7c34231108560dd"} Jul 23 11:26:50.702783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.699039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"dc18ff23db0a42fd1bee7e6bdb4fc45bfaad8d316ebd7fc227a987c65ca3bbb6"} Jul 23 11:26:50.702783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.702087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6glc" event={"ID":"d2044830-359c-4bb5-b06d-7080785eaa70","Type":"ContainerStarted","Data":"28cef93b1819c620eafc321623cbe993577eb8dcbbfef7bdd480e7def6c15e17"} Jul 23 11:26:50.709714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.709670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blnrj" event={"ID":"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61","Type":"ContainerStarted","Data":"5e039dfca040addb595c57b4dc83ef2bc17de47ae6c422f006eb7fbe72e27918"} Jul 23 11:26:50.713558 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:50.713400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" event={"ID":"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45","Type":"ContainerStarted","Data":"bf35c2b5479e129ca43a57a061eb2444c2bdce1893047f6897c482a10da5b45c"} Jul 23 11:26:51.537528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:51.537380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:51.537528 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:51.537505 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:51.538272 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:51.537900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:51.538272 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:51.538002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:51.736365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:51.736239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" event={"ID":"6f4e9d4dfd88fb1f06ac352c38a2cfb2","Type":"ContainerStarted","Data":"ad4b47ee50b53f5b4edfbc2f2e714dac5b24bfc4640e11ecf6f9e2c4fbc7f608"} Jul 23 11:26:51.753401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:51.753337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-30.ec2.internal" podStartSLOduration=4.753311872 podStartE2EDuration="4.753311872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:26:50.616292898 +0000 UTC m=+4.598290402" watchObservedRunningTime="2026-07-23 11:26:51.753311872 +0000 UTC m=+5.735309377" Jul 23 11:26:52.129600 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:52.129555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:52.129792 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:52.129632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:52.129792 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:52.129683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:52.129905 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.129857 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:52.129967 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.129914 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:26:56.129900954 +0000 UTC m=+10.111898435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:52.130314 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130294 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:52.130314 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130315 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:52.130460 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130325 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:52.130460 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130358 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:26:56.130348035 +0000 UTC m=+10.112345515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:52.130460 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130401 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:52.130460 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.130422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:26:56.130414118 +0000 UTC m=+10.112411599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:52.539332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:52.537582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:52.539332 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:52.537714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:53.537298 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:53.537260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:53.537464 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:53.537272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:53.537464 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:53.537404 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:53.537585 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:53.537467 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:54.538932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:54.538897 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:54.539355 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:54.539023 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:55.537004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:55.536969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:55.537184 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:55.537088 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:55.537460 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:55.537434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:55.537589 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:55.537547 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:56.167246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:56.167207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:56.167280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:56.167317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167344 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167403 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:27:04.167387686 +0000 UTC m=+18.149385167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167432 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167481 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:27:04.167465696 +0000 UTC m=+18.149463191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167569 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167582 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167593 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:56.167640 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.167630 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:04.167616795 +0000 UTC m=+18.149614279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:26:56.538914 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:56.538421 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:56.538914 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:56.538548 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:57.537152 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:57.537115 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:57.537642 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:57.537266 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:57.537642 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:57.537134 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:57.537789 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:57.537672 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:26:58.537186 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:58.537146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:26:58.537832 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:58.537283 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:26:59.537231 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:59.537192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:26:59.537721 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:59.537334 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:26:59.537721 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:26:59.537391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:26:59.537721 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:26:59.537496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:00.536901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:00.536867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:00.537068 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:00.536993 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:01.537286 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:01.537254 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:01.537744 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:01.537377 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:01.537744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:01.537407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:01.537744 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:01.537521 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:02.536646 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:02.536611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:02.536839 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:02.536726 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:03.536733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:03.536693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:03.537183 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:03.536747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:03.537183 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:03.536871 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:03.537183 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:03.536989 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:04.231734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:04.231689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:04.231747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:04.231799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231874 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231898 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231910 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231913 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:04.231939 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231913 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:04.232161 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231963 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:20.231949508 +0000 UTC m=+34.213946990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:04.232161 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231979 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:27:20.231971342 +0000 UTC m=+34.213968822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:04.232161 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.231989 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:27:20.231984093 +0000 UTC m=+34.213981574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:04.537229 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:04.537154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:04.537609 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:04.537282 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:05.536892 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:05.536859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:05.536892 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:05.536879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:05.537108 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:05.536996 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:05.537160 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:05.537112 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:06.537387 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:06.537347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:06.537789 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:06.537459 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:07.537319 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:07.537280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:07.537480 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:07.537280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:07.537480 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:07.537410 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:07.537903 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:07.537488 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:08.536653 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.536461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:08.536785 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:08.536736 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:08.773259 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.773111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" event={"ID":"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45","Type":"ContainerStarted","Data":"5643e01b6017c3135e5221804b2552df415f6cc01f9dbf343b5ded533833398c"} Jul 23 11:27:08.774466 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.774447 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="ccdc70d915aa23abe227325e12040b27ac7d40d1d0ec2ffb342fe3ee662bb85d" exitCode=0 Jul 23 11:27:08.774551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.774509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"ccdc70d915aa23abe227325e12040b27ac7d40d1d0ec2ffb342fe3ee662bb85d"} Jul 23 11:27:08.776027 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.776010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6xkh9" event={"ID":"7655ae45-899d-45ed-a10f-6286660f2c51","Type":"ContainerStarted","Data":"31a3c9206986be50fc8a2b029a56f28e9dae18c899bb7959913f35bf9fa3f864"} Jul 23 11:27:08.777390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.777366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs9l8" event={"ID":"04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16","Type":"ContainerStarted","Data":"0ba0e81159848b57f3a4f327c998551509f5551ed0cb405b0554519d5e822a7b"} Jul 23 11:27:08.778820 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.778790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96klp" event={"ID":"01887123-dc9b-428d-967f-8d195ad04afe","Type":"ContainerStarted","Data":"2b802566c7ecaa24e89d818a48d512f9a7943fc1cf8543f3ac616cbaa509d66e"} Jul 23 11:27:08.781832 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.781811 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:27:08.782199 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.782172 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ecbc862-e2ba-430c-9bab-dfcb931bc27b" containerID="60a430e8182c50987c25fbfc124397a288c96903822badc8723f91680fe67f4e" exitCode=1 Jul 23 11:27:08.782284 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.782242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"a54331977c954bf88b9817fd187929add970d2c36f45cfcced0c395db4fc669e"} Jul 23 11:27:08.782343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.782326 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"9cad466e4ff973a4f29c83fda0c24d2a263c8b7a3986b279ad4532b7d98a71a8"} Jul 23 11:27:08.782396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.782380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerDied","Data":"60a430e8182c50987c25fbfc124397a288c96903822badc8723f91680fe67f4e"} Jul 23 11:27:08.782456 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.782428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"054840c57aea9e1dbfd1c583c155fd1dcd80b5ee6a5494812de6eb0a2d6759b0"} Jul 23 11:27:08.783809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.783789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6glc" event={"ID":"d2044830-359c-4bb5-b06d-7080785eaa70","Type":"ContainerStarted","Data":"b4188c94357358e208eb7d8d5be8674dd3ff84d2f8787e642ff8d2ab3eaf3276"} Jul 23 11:27:08.785357 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.785327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blnrj" event={"ID":"27b1670e-e6b4-4904-ad2e-a8ed9e7a8d61","Type":"ContainerStarted","Data":"f23adc04b9be5301bdb3742e3b6afa9979377e9703456c636166f05d7b806602"} Jul 23 11:27:08.798688 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.798653 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-30.ec2.internal" podStartSLOduration=21.798641615 podStartE2EDuration="21.798641615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:26:51.753673525 +0000 UTC m=+5.735671030" watchObservedRunningTime="2026-07-23 11:27:08.798641615 +0000 UTC m=+22.780639118" Jul 23 11:27:08.811045 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.811014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bs9l8" podStartSLOduration=4.29015681 podStartE2EDuration="22.811002276s" totalImagesPullingTime="18.520845466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:08.810701901 +0000 UTC m=+22.792699405" watchObservedRunningTime="2026-07-23 11:27:08.811002276 +0000 UTC m=+22.792999775" Jul 23 11:27:08.826855 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.826808 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-blnrj" podStartSLOduration=4.302749621 podStartE2EDuration="22.82679637s" totalImagesPullingTime="18.524046749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:08.826747959 +0000 UTC m=+22.808745463" watchObservedRunningTime="2026-07-23 11:27:08.82679637 +0000 UTC m=+22.808793872" Jul 23 11:27:08.840499 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.840467 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6xkh9" podStartSLOduration=4.357106348 podStartE2EDuration="22.840457321s" totalImagesPullingTime="18.483350973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:08.840355732 +0000 UTC m=+22.822353235" watchObservedRunningTime="2026-07-23 11:27:08.840457321 +0000 UTC m=+22.822454821" Jul 23 11:27:08.853886 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.853854 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-96klp" podStartSLOduration=4.372527203 podStartE2EDuration="22.853843674s" totalImagesPullingTime="18.481316471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:08.853366772 +0000 UTC m=+22.835364275" watchObservedRunningTime="2026-07-23 11:27:08.853843674 +0000 UTC m=+22.835841213" Jul 23 11:27:08.870881 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:08.870846 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l6glc" podStartSLOduration=4.07024845 podStartE2EDuration="22.870835805s" totalImagesPullingTime="18.800587355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:08.870830115 +0000 UTC m=+22.852827618" watchObservedRunningTime="2026-07-23 11:27:08.870835805 +0000 UTC m=+22.852833308" Jul 23 11:27:09.023869 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.023828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:09.095223 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.095186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:09.536573 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.536546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:09.536700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.536563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:09.536700 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:09.536672 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:09.536798 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:09.536717 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:09.563922 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.563896 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 11:27:09.794240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.794207 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:27:09.795037 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.795010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"59c987d3293cff2672570e1996e9eb23ac75f38b8c496b13d61f8ae3c40f145a"} Jul 23 11:27:09.795175 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.795121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"722a61929ceed257c9b980d5e058e96cd17f20632438c7ad0a7d3e75378d5d8a"} Jul 23 11:27:09.796977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.796952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" event={"ID":"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45","Type":"ContainerStarted","Data":"ce56b0a1b0208726473a6c1890dd1ff8f087bb75f8ba824db87e5138f72af88b"} Jul 23 11:27:09.798430 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.798405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-85jmq" event={"ID":"30f59c66-9b75-4cc3-aea4-020344e4d396","Type":"ContainerStarted","Data":"39590f058db2afa0ec65102a3898da609eba7404ded1e29b3faa6bfab197fda2"} Jul 23 11:27:09.948416 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:09.948381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:10.046019 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.045942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:10.449506 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.449346 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:27:09.563916042Z","UUID":"2ed0d9a2-3a47-48ea-844f-ae145d6d8929","Handler":null,"Name":"","Endpoint":""} Jul 23 11:27:10.450932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.450907 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 11:27:10.451065 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.450955 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 11:27:10.537503 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.537473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:10.537667 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:10.537608 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:10.803313 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.803277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" event={"ID":"af9dbea7-fb82-4488-aa5e-3b1aa99ddf45","Type":"ContainerStarted","Data":"130db6cd90ba3b40e5c67226a14b7777a651cd8be50668589c9627ab16f7af49"} Jul 23 11:27:10.821024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.820971 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6n5l" podStartSLOduration=4.2360700940000005 podStartE2EDuration="24.820950568s" totalImagesPullingTime="20.584880474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:10.820789593 +0000 UTC m=+24.802787096" watchObservedRunningTime="2026-07-23 11:27:10.820950568 +0000 UTC m=+24.802948073" Jul 23 11:27:10.821339 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:10.821308 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-85jmq" podStartSLOduration=6.305328851 podStartE2EDuration="24.821298866s" totalImagesPullingTime="18.515970015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:09.81347555 +0000 UTC m=+23.795473064" watchObservedRunningTime="2026-07-23 11:27:10.821298866 +0000 UTC m=+24.803296368" Jul 23 11:27:11.537030 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.536868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:11.537181 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.536922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:11.537181 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:11.537101 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:11.537245 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:11.537179 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:11.723245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.723143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:11.808822 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.808740 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:27:11.809246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.809221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"ad20ea02868b201b8301ea91d7928b5721c0b5763b01d23c013c86a3c8d2787a"} Jul 23 11:27:11.820553 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.820522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:11.839755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.839728 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:27:11.840327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.840307 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:27:11.920131 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:11.920090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:12.021022 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.020977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:12.537432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.537403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:12.537624 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:12.537529 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:12.760440 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.760398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:12.811666 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.811588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:27:12.812256 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.812229 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-96klp" Jul 23 11:27:12.866425 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.866387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:12.931661 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:12.931627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.169250 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.169161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.233936 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.233906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.489755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.489587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.536747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.536719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:13.536747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.536749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:13.536963 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:13.536854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:13.536963 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:13.536930 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:13.573352 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.573323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.712126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.712087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.772695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.772603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.815028 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.814995 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="e7ab31e54710eb24666aff274f9d994ae0bfcbdcb67ce04320790acd70d782cc" exitCode=0 Jul 23 11:27:13.815691 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.815088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"e7ab31e54710eb24666aff274f9d994ae0bfcbdcb67ce04320790acd70d782cc"} Jul 23 11:27:13.818361 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.818345 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:27:13.818774 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.818727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"c52f420fba79403e409790266e8eceb2a57f3c7b8fb41e710f22aa42d1b97350"} Jul 23 11:27:13.819169 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.819154 2564 scope.go:117] "RemoveContainer" containerID="60a430e8182c50987c25fbfc124397a288c96903822badc8723f91680fe67f4e" Jul 23 11:27:13.880791 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.880738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:13.936198 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:13.936159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:14.203227 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.203153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:14.231706 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.231682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6xkh9_7655ae45-899d-45ed-a10f-6286660f2c51/dns-node-resolver/0.log" Jul 23 11:27:14.302826 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.302790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:14.537343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.537313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:14.537551 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:14.537424 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:14.695338 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.695301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:14.825076 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.825018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:27:14.825445 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.825406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" event={"ID":"7ecbc862-e2ba-430c-9bab-dfcb931bc27b","Type":"ContainerStarted","Data":"7b917d5f1e172e81709f02bb7ff5c6316dab43d57a24b0f4af49730ff24a6ee7"} Jul 23 11:27:14.825549 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.825531 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 11:27:14.825858 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.825839 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:14.825943 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.825911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:14.842004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.841975 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:14.842170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.842108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:14.853167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:14.853121 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" podStartSLOduration=10.276404929 podStartE2EDuration="28.853105887s" totalImagesPullingTime="18.576700958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:14.851646528 +0000 UTC m=+28.833644030" watchObservedRunningTime="2026-07-23 11:27:14.853105887 +0000 UTC m=+28.835103390" Jul 23 11:27:15.233039 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.232881 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs9l8_04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16/node-ca/0.log" Jul 23 11:27:15.292335 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.292299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qqn65"] Jul 23 11:27:15.292472 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.292417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:15.292568 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:15.292518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:15.295880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.295838 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dfxr"] Jul 23 11:27:15.296009 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.295967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:15.296093 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:15.296063 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:15.296453 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.296422 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfvsb"] Jul 23 11:27:15.296563 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.296524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:15.296627 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:15.296607 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:15.610577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.610528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:15.669615 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.669573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:15.794405 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.794366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:15.830185 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.830152 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="b4d6623487638f65618ceae7b831544dcd55104d7253904616e9af94c6593082" exitCode=0 Jul 23 11:27:15.830695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.830230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"b4d6623487638f65618ceae7b831544dcd55104d7253904616e9af94c6593082"} Jul 23 11:27:15.830695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.830280 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 11:27:15.857985 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:15.857957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:16.537856 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:16.537824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:16.538003 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:16.537921 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:16.832520 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:16.832445 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 11:27:17.001653 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.001617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.146870 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.146847 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:17.268194 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.268158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.467024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.466936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.536551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.536524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:17.536684 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.536525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:17.536718 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:17.536619 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:17.536718 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:17.536701 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:17.579207 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.579178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.711273 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.711234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.836970 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.836929 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="e51e6cc61a375e705460f780c4db9c1d3a6420f50ad1deec6cf44f024fda42a0" exitCode=0 Jul 23 11:27:17.836970 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.836964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"e51e6cc61a375e705460f780c4db9c1d3a6420f50ad1deec6cf44f024fda42a0"} Jul 23 11:27:17.894577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.894543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:17.997323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:17.997286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:18.199337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:18.199258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:18.542028 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:18.542003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:18.542161 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:18.542105 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:18.943201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:18.943085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:19.342683 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.342646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:19.536483 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.536454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:19.536659 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:19.536575 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:19.536902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.536459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:19.537027 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:19.536989 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:19.541239 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.541209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:19.742232 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.742195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:19.940455 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:19.940420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:20.257211 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.257176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.257230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257333 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.257363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257366 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257389 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs podName:bd6840b5-931b-4a08-8db0-f068a43d2a8d nodeName:}" failed. No retries permitted until 2026-07-23 11:27:52.257371426 +0000 UTC m=+66.239368907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs") pod "network-metrics-daemon-qqn65" (UID: "bd6840b5-931b-4a08-8db0-f068a43d2a8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257397 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257409 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d4wp5 for pod openshift-network-diagnostics/network-check-target-2dfxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257445 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5 podName:1fd62e54-3990-4dd7-9ae3-ac8cb44368f4 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:52.25743575 +0000 UTC m=+66.239433245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4wp5" (UniqueName: "kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5") pod "network-check-target-2dfxr" (UID: "1fd62e54-3990-4dd7-9ae3-ac8cb44368f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257461 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:20.257655 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.257519 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret podName:ab9726ae-8666-45a9-89d2-91e32363b1ee nodeName:}" failed. No retries permitted until 2026-07-23 11:27:52.257500364 +0000 UTC m=+66.239497858 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret") pod "global-pull-secret-syncer-hfvsb" (UID: "ab9726ae-8666-45a9-89d2-91e32363b1ee") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:27:20.536746 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.536706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:20.536942 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:20.536851 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:20.799438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.799193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:20.945104 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:20.945072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.008488 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.008454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.338486 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.338457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.537044 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.537009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:21.537277 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:21.537138 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:21.537512 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.537015 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:21.537614 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:21.537593 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:21.548461 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.548428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.766370 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.766336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.868571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.868529 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:21.969623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:21.969586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:22.537467 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:22.537435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:22.537879 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:22.537548 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:22.852849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:22.852751 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="4212f4a7da4ede06c3b8bd50d13e83b0a359f771b0f9f1059b2bb4319db4864d" exitCode=0 Jul 23 11:27:22.852849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:22.852811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"4212f4a7da4ede06c3b8bd50d13e83b0a359f771b0f9f1059b2bb4319db4864d"} Jul 23 11:27:23.021291 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.021256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:23.339289 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.339260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:23.448881 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.448834 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:23.536898 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.536862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:23.536898 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.536900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:23.537090 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:23.536975 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:23.537127 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:23.537100 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:23.548873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.548841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:23.741068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.741037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:23.858195 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.858163 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e83780-8bf1-4ece-b2da-a4024c2576b2" containerID="e6a70e2d83eef401affd1db16f9f7aa5c72004a99da257ce90d9e3d437e3dbaf" exitCode=0 Jul 23 11:27:23.858377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.858231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerDied","Data":"e6a70e2d83eef401affd1db16f9f7aa5c72004a99da257ce90d9e3d437e3dbaf"} Jul 23 11:27:23.938278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:23.938251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:24.537460 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:24.537430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:24.537650 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:24.537558 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:24.863868 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:24.863831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" event={"ID":"89e83780-8bf1-4ece-b2da-a4024c2576b2","Type":"ContainerStarted","Data":"54f751319be6accdde2b1fb879fbd9df65b033593234df05dccdae53d1c7e8ca"} Jul 23 11:27:24.887382 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:24.887340 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zv2tw" podStartSLOduration=9.44540047 podStartE2EDuration="38.887327082s" totalImagesPullingTime="29.441926612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:24.887261538 +0000 UTC m=+38.869259053" watchObservedRunningTime="2026-07-23 11:27:24.887327082 +0000 UTC m=+38.869324587" Jul 23 11:27:24.951879 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:24.951852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:25.335159 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.335129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:25.537159 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.537128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:25.537330 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:25.537213 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:25.537330 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.537128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:25.537330 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:25.537320 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:25.600418 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.600343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:25.738406 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.738372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:25.936602 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:25.936515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:26.213613 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:26.213543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:26.388986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:26.388957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:26.537902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:26.537875 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:26.538064 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:26.537976 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:26.940896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:26.940784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:27.339443 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.339406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:27.537020 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.536990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:27.537192 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.536990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:27.537192 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:27.537086 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:27.537192 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:27.537160 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:27.547756 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.547720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:27.742521 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.742488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:27.940457 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:27.940422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:28.276614 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:28.276580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:28.536911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:28.536833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:28.537054 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:28.536943 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:29.537239 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:29.537203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:29.537616 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:29.537203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:29.537616 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:29.537308 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:29.537616 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:29.537374 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:30.537090 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:30.537056 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:30.537264 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:30.537164 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:30.612744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:30.612698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:30.713014 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:30.712980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:31.537427 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:31.537390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:31.537427 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:31.537429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:31.537880 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:31.537509 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:31.537880 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:31.537618 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:32.539662 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:32.539636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:32.540051 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:32.539730 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:33.537123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:33.537092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:33.537123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:33.537131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:33.537335 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:33.537221 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:33.537373 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:33.537327 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:34.536921 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:34.536888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:34.537307 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:34.536999 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:35.536728 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:35.536693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:35.537015 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:35.536692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:35.537015 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:35.536814 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dfxr" podUID="1fd62e54-3990-4dd7-9ae3-ac8cb44368f4" Jul 23 11:27:35.537015 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:35.536861 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfvsb" podUID="ab9726ae-8666-45a9-89d2-91e32363b1ee" Jul 23 11:27:35.844561 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:35.844475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:35.900651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:35.900621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.003071 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.003039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.113244 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.113164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.289035 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.289004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.416923 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.416827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.537549 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.537520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:36.538017 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:36.537640 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qqn65" podUID="bd6840b5-931b-4a08-8db0-f068a43d2a8d" Jul 23 11:27:36.571468 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.571441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.692982 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.692892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.787069 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.787037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:36.984245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:36.984213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.063122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.063088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.131266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.131233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.312133 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.312059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.402494 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.402463 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-30.ec2.internal" event="NodeReady" Jul 23 11:27:37.402658 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.402587 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 11:27:37.443805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.443751 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dd4b8fc4b-75vcl"] Jul 23 11:27:37.479242 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.479211 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m4xtr"] Jul 23 11:27:37.479527 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.479330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.482745 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.482092 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 11:27:37.482745 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.482239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t2g4d\"" Jul 23 11:27:37.482745 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.482483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 11:27:37.482745 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.482546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 11:27:37.492976 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.492955 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd4b8fc4b-75vcl"] Jul 23 11:27:37.493061 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.492995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cdztf"] Jul 23 11:27:37.493113 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.493095 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.495422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.495401 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6qnx\"" Jul 23 11:27:37.495538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.495424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 11:27:37.495538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.495406 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 11:27:37.498950 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.498928 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 11:27:37.507240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.507220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cdztf"] Jul 23 11:27:37.507350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.507254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4xtr"] Jul 23 11:27:37.507350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.507337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.509688 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.509664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 11:27:37.509827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.509699 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 11:27:37.509827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.509709 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 11:27:37.509827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.509723 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfh54\"" Jul 23 11:27:37.509827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.509804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 11:27:37.523002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.522964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.536383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.536358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:37.536513 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.536358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:37.538702 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.538681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:27:37.538702 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.538698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:27:37.539113 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.538681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8j6tk\"" Jul 23 11:27:37.539113 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.538681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:27:37.557965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.557936 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rmglm"] Jul 23 11:27:37.571264 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.571201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.574068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.574048 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 11:27:37.574230 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.574213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89ttz\"" Jul 23 11:27:37.574307 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.574287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rmglm"] Jul 23 11:27:37.574673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.574654 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 11:27:37.574819 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.574733 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 11:27:37.587530 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84ww\" (UniqueName: \"kubernetes.io/projected/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-api-access-d84ww\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.587645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f5bc939-1678-41f0-9073-3a82fbaa94ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.587645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6qc\" (UniqueName: \"kubernetes.io/projected/507d5574-83f2-439f-9c08-bbcb51b1be6f-kube-api-access-jn6qc\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.587645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsx4v\" (UniqueName: \"kubernetes.io/projected/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-kube-api-access-nsx4v\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.587747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-tls\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-ca-trust-extracted\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-cert\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.587747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507d5574-83f2-439f-9c08-bbcb51b1be6f-tmp-dir\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzns\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-kube-api-access-4wzns\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-certificates\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-bound-sa-token\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-trusted-ca\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/507d5574-83f2-439f-9c08-bbcb51b1be6f-metrics-tls\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f5bc939-1678-41f0-9073-3a82fbaa94ed-crio-socket\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-installation-pull-secrets\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.587915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f5bc939-1678-41f0-9073-3a82fbaa94ed-data-volume\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.588230 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.587968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-image-registry-private-configuration\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.588230 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.588002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/507d5574-83f2-439f-9c08-bbcb51b1be6f-config-volume\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.623080 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.623047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:37.688295 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsx4v\" (UniqueName: \"kubernetes.io/projected/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-kube-api-access-nsx4v\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.688435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-tls\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-ca-trust-extracted\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-cert\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.688435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507d5574-83f2-439f-9c08-bbcb51b1be6f-tmp-dir\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.688435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzns\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-kube-api-access-4wzns\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-certificates\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-bound-sa-token\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-trusted-ca\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/507d5574-83f2-439f-9c08-bbcb51b1be6f-metrics-tls\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f5bc939-1678-41f0-9073-3a82fbaa94ed-crio-socket\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.688725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-installation-pull-secrets\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f5bc939-1678-41f0-9073-3a82fbaa94ed-data-volume\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-image-registry-private-configuration\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/507d5574-83f2-439f-9c08-bbcb51b1be6f-config-volume\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507d5574-83f2-439f-9c08-bbcb51b1be6f-tmp-dir\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84ww\" (UniqueName: \"kubernetes.io/projected/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-api-access-d84ww\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-ca-trust-extracted\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f5bc939-1678-41f0-9073-3a82fbaa94ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.689100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.688939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6qc\" (UniqueName: \"kubernetes.io/projected/507d5574-83f2-439f-9c08-bbcb51b1be6f-kube-api-access-jn6qc\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.689476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.689238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.689476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.689449 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/507d5574-83f2-439f-9c08-bbcb51b1be6f-config-volume\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.689576 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.689452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-certificates\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.690466 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.690406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f5bc939-1678-41f0-9073-3a82fbaa94ed-data-volume\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.691130 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.691104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f5bc939-1678-41f0-9073-3a82fbaa94ed-crio-socket\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.691517 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.691491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-trusted-ca\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.695053 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.693665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-installation-pull-secrets\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.695053 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.693854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f5bc939-1678-41f0-9073-3a82fbaa94ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.695053 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.694118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-cert\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.695450 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.695429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/507d5574-83f2-439f-9c08-bbcb51b1be6f-metrics-tls\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.696099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.696074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-image-registry-private-configuration\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.696557 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.696532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-registry-tls\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.697979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.697944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzns\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-kube-api-access-4wzns\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.698079 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.698000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b-bound-sa-token\") pod \"image-registry-7dd4b8fc4b-75vcl\" (UID: \"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b\") " pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.698142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.698115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsx4v\" (UniqueName: \"kubernetes.io/projected/a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd-kube-api-access-nsx4v\") pod \"ingress-canary-rmglm\" (UID: \"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd\") " pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:37.698198 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.698156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6qc\" (UniqueName: \"kubernetes.io/projected/507d5574-83f2-439f-9c08-bbcb51b1be6f-kube-api-access-jn6qc\") pod \"dns-default-m4xtr\" (UID: \"507d5574-83f2-439f-9c08-bbcb51b1be6f\") " pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.698550 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.698531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84ww\" (UniqueName: \"kubernetes.io/projected/6f5bc939-1678-41f0-9073-3a82fbaa94ed-kube-api-access-d84ww\") pod \"insights-runtime-extractor-cdztf\" (UID: \"6f5bc939-1678-41f0-9073-3a82fbaa94ed\") " pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.793538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.793492 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:37.801677 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.801643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:37.815846 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.815820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cdztf" Jul 23 11:27:37.881196 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:37.881152 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rmglm" Jul 23 11:27:38.029024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.028909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd4b8fc4b-75vcl"] Jul 23 11:27:38.036272 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:38.036244 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8436097c_ebd7_4aa1_b7f0_c4cc7b056a5b.slice/crio-e17cbf377504fe18ab7da1a073d947350cf56bbf2ab2be32a3a2788f9f7aab81 WatchSource:0}: Error finding container e17cbf377504fe18ab7da1a073d947350cf56bbf2ab2be32a3a2788f9f7aab81: Status 404 returned error can't find the container with id e17cbf377504fe18ab7da1a073d947350cf56bbf2ab2be32a3a2788f9f7aab81 Jul 23 11:27:38.062730 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.062597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rmglm"] Jul 23 11:27:38.065153 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:38.065125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d788b9_c6ed_46fd_81f9_89ac7de3a1dd.slice/crio-a0088756eb97fdf848ed70a8f084a2d019219ac9a5853ca48caa817bedd05666 WatchSource:0}: Error finding container a0088756eb97fdf848ed70a8f084a2d019219ac9a5853ca48caa817bedd05666: Status 404 returned error can't find the container with id a0088756eb97fdf848ed70a8f084a2d019219ac9a5853ca48caa817bedd05666 Jul 23 11:27:38.172743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.172689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:38.237420 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.237383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cdztf"] Jul 23 11:27:38.240604 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.240565 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4xtr"] Jul 23 11:27:38.242777 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:38.242735 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5bc939_1678_41f0_9073_3a82fbaa94ed.slice/crio-3d21b64f2740b28e0c6852bdf800196175145740cbcf7b017c57157719b920a8 WatchSource:0}: Error finding container 3d21b64f2740b28e0c6852bdf800196175145740cbcf7b017c57157719b920a8: Status 404 returned error can't find the container with id 3d21b64f2740b28e0c6852bdf800196175145740cbcf7b017c57157719b920a8 Jul 23 11:27:38.243452 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:38.243429 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507d5574_83f2_439f_9c08_bbcb51b1be6f.slice/crio-ee0b93ea80f7c678a62e0b157a8353e857fed7ce8ac2bea679d6a9aeae3ea49f WatchSource:0}: Error finding container ee0b93ea80f7c678a62e0b157a8353e857fed7ce8ac2bea679d6a9aeae3ea49f: Status 404 returned error can't find the container with id ee0b93ea80f7c678a62e0b157a8353e857fed7ce8ac2bea679d6a9aeae3ea49f Jul 23 11:27:38.521467 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.521423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:38.537421 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.537391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:38.539994 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.539973 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:27:38.540555 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.539975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zh6tm\"" Jul 23 11:27:38.574777 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.574725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:38.901818 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.901618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4xtr" event={"ID":"507d5574-83f2-439f-9c08-bbcb51b1be6f","Type":"ContainerStarted","Data":"ee0b93ea80f7c678a62e0b157a8353e857fed7ce8ac2bea679d6a9aeae3ea49f"} Jul 23 11:27:38.903963 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.903345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" event={"ID":"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b","Type":"ContainerStarted","Data":"b0c47c674173516301f32cf279df954b99f3073589bb7162ecf3b304d852c49c"} Jul 23 11:27:38.903963 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.903379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" event={"ID":"8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b","Type":"ContainerStarted","Data":"e17cbf377504fe18ab7da1a073d947350cf56bbf2ab2be32a3a2788f9f7aab81"} Jul 23 11:27:38.903963 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.903853 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:27:38.907474 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.907371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdztf" event={"ID":"6f5bc939-1678-41f0-9073-3a82fbaa94ed","Type":"ContainerStarted","Data":"3d21b64f2740b28e0c6852bdf800196175145740cbcf7b017c57157719b920a8"} Jul 23 11:27:38.909465 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.909437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rmglm" event={"ID":"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd","Type":"ContainerStarted","Data":"a0088756eb97fdf848ed70a8f084a2d019219ac9a5853ca48caa817bedd05666"} Jul 23 11:27:38.922896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:38.922839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" podStartSLOduration=2.922819146 podStartE2EDuration="2.922819146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:38.92129662 +0000 UTC m=+52.903294123" watchObservedRunningTime="2026-07-23 11:27:38.922819146 +0000 UTC m=+52.904816650" Jul 23 11:27:41.517585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:41.517551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:43.965909 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:43.965873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:44.066253 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:44.066216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:44.243504 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:44.243468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:44.340718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:44.340685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:44.446867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:44.446834 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:44.608253 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:44.608163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:45.017410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.017362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:45.117356 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.117327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:45.613086 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.613050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:45.682224 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.682178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:45.930961 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.930866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rmglm" event={"ID":"a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd","Type":"ContainerStarted","Data":"ab06bdc18ba764b25d268f7fbc45ca6ee0f52fa8f4ad99e45e52c41ec8812595"} Jul 23 11:27:45.932943 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.932912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4xtr" event={"ID":"507d5574-83f2-439f-9c08-bbcb51b1be6f","Type":"ContainerStarted","Data":"172ec038f01a225f393585561371bef4207fec325fb005e3f8f70ddd63ec8029"} Jul 23 11:27:45.933098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.932976 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:45.933098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.933004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4xtr" event={"ID":"507d5574-83f2-439f-9c08-bbcb51b1be6f","Type":"ContainerStarted","Data":"ac77741fe9e15bbdb751c7b90a09e28c5cd44be71a0f802a133122fc8d311874"} Jul 23 11:27:45.934523 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.934501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdztf" event={"ID":"6f5bc939-1678-41f0-9073-3a82fbaa94ed","Type":"ContainerStarted","Data":"a85c678b1c91890bb5502d2168a995f9d5301c9093195cb27dde336380aad452"} Jul 23 11:27:45.948324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.948283 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rmglm" podStartSLOduration=2.135038861 podStartE2EDuration="8.948270015s" totalImagesPullingTime="6.813231154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:45.947656789 +0000 UTC m=+59.929654291" watchObservedRunningTime="2026-07-23 11:27:45.948270015 +0000 UTC m=+59.930267496" Jul 23 11:27:45.966096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:45.966034 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m4xtr" podStartSLOduration=2.330975778 podStartE2EDuration="8.966010489s" totalImagesPullingTime="6.635034711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:45.964776247 +0000 UTC m=+59.946773754" watchObservedRunningTime="2026-07-23 11:27:45.966010489 +0000 UTC m=+59.948007993" Jul 23 11:27:47.849235 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:47.849208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4kjj5" Jul 23 11:27:47.988871 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:47.988837 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:48.050523 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:48.050489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:48.105734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:48.105655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:48.946146 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:48.946110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdztf" event={"ID":"6f5bc939-1678-41f0-9073-3a82fbaa94ed","Type":"ContainerStarted","Data":"5a81a3c8f4443347b3d7b17a9fd1f5ecfbcb43074fe244fb55dfe1715a154445"} Jul 23 11:27:49.785195 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:49.785158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:49.885452 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:49.885412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:50.002874 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:50.002835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:50.102595 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:50.102512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:50.486550 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:50.486513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:50.555885 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:50.555847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:50.659308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:50.659270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:51.019827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:51.019792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:51.121400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:51.121365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:52.303002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.302969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:52.303442 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.303025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:52.303442 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.303051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:52.306013 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.305981 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:27:52.306013 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.305984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:27:52.306503 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.306489 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:27:52.316134 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.316109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab9726ae-8666-45a9-89d2-91e32363b1ee-original-pull-secret\") pod \"global-pull-secret-syncer-hfvsb\" (UID: \"ab9726ae-8666-45a9-89d2-91e32363b1ee\") " pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:52.316217 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.316153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6840b5-931b-4a08-8db0-f068a43d2a8d-metrics-certs\") pod \"network-metrics-daemon-qqn65\" (UID: \"bd6840b5-931b-4a08-8db0-f068a43d2a8d\") " pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:52.316413 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.316397 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:27:52.325941 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.325918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wp5\" (UniqueName: \"kubernetes.io/projected/1fd62e54-3990-4dd7-9ae3-ac8cb44368f4-kube-api-access-d4wp5\") pod \"network-check-target-2dfxr\" (UID: \"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4\") " pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:52.348615 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.348592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zh6tm\"" Jul 23 11:27:52.356607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.356589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qqn65" Jul 23 11:27:52.483081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.483048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qqn65"] Jul 23 11:27:52.486017 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:52.485992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6840b5_931b_4a08_8db0_f068a43d2a8d.slice/crio-3b0c9adb647ddf23f86005891d9c4761be2effabb45a06a428ee6ae401dff40a WatchSource:0}: Error finding container 3b0c9adb647ddf23f86005891d9c4761be2effabb45a06a428ee6ae401dff40a: Status 404 returned error can't find the container with id 3b0c9adb647ddf23f86005891d9c4761be2effabb45a06a428ee6ae401dff40a Jul 23 11:27:52.548192 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.548163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8j6tk\"" Jul 23 11:27:52.550130 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.550116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfvsb" Jul 23 11:27:52.556056 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.556010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:52.674558 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.674489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfvsb"] Jul 23 11:27:52.677844 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:52.677820 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9726ae_8666_45a9_89d2_91e32363b1ee.slice/crio-5de3c3714b5b7677652bc34038da4f220024e9791e53408852448b554f0941f0 WatchSource:0}: Error finding container 5de3c3714b5b7677652bc34038da4f220024e9791e53408852448b554f0941f0: Status 404 returned error can't find the container with id 5de3c3714b5b7677652bc34038da4f220024e9791e53408852448b554f0941f0 Jul 23 11:27:52.689818 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.689795 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dfxr"] Jul 23 11:27:52.692745 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:52.692722 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd62e54_3990_4dd7_9ae3_ac8cb44368f4.slice/crio-bc752843d2fd3f4152b0e00c2d538f32575e58b95b985c60f8627b08b90589a1 WatchSource:0}: Error finding container bc752843d2fd3f4152b0e00c2d538f32575e58b95b985c60f8627b08b90589a1: Status 404 returned error can't find the container with id bc752843d2fd3f4152b0e00c2d538f32575e58b95b985c60f8627b08b90589a1 Jul 23 11:27:52.961515 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.961430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdztf" event={"ID":"6f5bc939-1678-41f0-9073-3a82fbaa94ed","Type":"ContainerStarted","Data":"5709d8ebee499d5b90c008446e1ccb0857de0c013fc9fc5d445856cdc85b6b26"} Jul 23 11:27:52.962738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.962709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfvsb" event={"ID":"ab9726ae-8666-45a9-89d2-91e32363b1ee","Type":"ContainerStarted","Data":"5de3c3714b5b7677652bc34038da4f220024e9791e53408852448b554f0941f0"} Jul 23 11:27:52.964009 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.963986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qqn65" event={"ID":"bd6840b5-931b-4a08-8db0-f068a43d2a8d","Type":"ContainerStarted","Data":"3b0c9adb647ddf23f86005891d9c4761be2effabb45a06a428ee6ae401dff40a"} Jul 23 11:27:52.965256 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.965237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dfxr" event={"ID":"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4","Type":"ContainerStarted","Data":"bc752843d2fd3f4152b0e00c2d538f32575e58b95b985c60f8627b08b90589a1"} Jul 23 11:27:52.979375 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:52.979331 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cdztf" podStartSLOduration=1.975151435 podStartE2EDuration="15.97931237s" totalImagesPullingTime="14.004160935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:27:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:52.978508607 +0000 UTC m=+66.960506111" watchObservedRunningTime="2026-07-23 11:27:52.97931237 +0000 UTC m=+66.961309876" Jul 23 11:27:53.971532 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:53.971427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qqn65" event={"ID":"bd6840b5-931b-4a08-8db0-f068a43d2a8d","Type":"ContainerStarted","Data":"6066bc91a51f47c9c0cadc7dd0667dcae0537bcf9903ddf2c320a34a5a5668ec"} Jul 23 11:27:53.971532 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:53.971486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qqn65" event={"ID":"bd6840b5-931b-4a08-8db0-f068a43d2a8d","Type":"ContainerStarted","Data":"d23b7728e9ffe8cd616ca0f8c7bdb72df6515233def7eef1cb1da2ee05a71ba1"} Jul 23 11:27:53.993533 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:53.993474 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qqn65" podStartSLOduration=67.00587328 podStartE2EDuration="1m7.993448567s" totalImagesPullingTime="987.575287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:53.991886549 +0000 UTC m=+67.973884052" watchObservedRunningTime="2026-07-23 11:27:53.993448567 +0000 UTC m=+67.975446071" Jul 23 11:27:54.121505 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:54.121453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:54.221322 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:54.221274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:54.528226 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:54.528194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:55.941203 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:55.941174 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m4xtr" Jul 23 11:27:56.244284 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:56.244242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:56.981539 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:56.981496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:57.083241 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.083195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:57.138329 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.138225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:57.797905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.797819 2564 patch_prober.go:28] interesting pod/image-registry-7dd4b8fc4b-75vcl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 11:27:57.797905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.797873 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" podUID="8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 11:27:57.987977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.987946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dfxr" event={"ID":"1fd62e54-3990-4dd7-9ae3-ac8cb44368f4","Type":"ContainerStarted","Data":"60bcb604475034568b7f63e3ecd5e577ee6920e617145d3884bcd504c7b7896a"} Jul 23 11:27:57.988412 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.988039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:27:57.989629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:57.989605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfvsb" event={"ID":"ab9726ae-8666-45a9-89d2-91e32363b1ee","Type":"ContainerStarted","Data":"3663ea9ef1b6383733eae8c1040eef28478879c8e99d75a241470798b64f3767"} Jul 23 11:27:58.005391 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.005347 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2dfxr" podStartSLOduration=67.205923651 podStartE2EDuration="1m12.005331999s" totalImagesPullingTime="4.799408348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:58.004331316 +0000 UTC m=+71.986328819" watchObservedRunningTime="2026-07-23 11:27:58.005331999 +0000 UTC m=+71.987329634" Jul 23 11:27:58.628722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.628685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:58.848754 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.848723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:58.908310 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.908197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hfvsb" podStartSLOduration=68.092028804 podStartE2EDuration="1m12.908160312s" totalImagesPullingTime="4.816131508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:27:58.019712833 +0000 UTC m=+72.001710347" watchObservedRunningTime="2026-07-23 11:27:58.908160312 +0000 UTC m=+72.890157814" Jul 23 11:27:58.908605 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.908589 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mf675"] Jul 23 11:27:58.910670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.910649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.913922 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.913903 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 11:27:58.914697 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.914682 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 11:27:58.914778 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.914732 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 11:27:58.917245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.917229 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 11:27:58.917970 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.917953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 11:27:58.918077 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.917989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 11:27:58.918077 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.917996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2zc7r\"" Jul 23 11:27:58.921165 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.921147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8fsl4"] Jul 23 11:27:58.923266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.923251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.925934 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.925916 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m86hz\"" Jul 23 11:27:58.926039 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.925918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 11:27:58.926039 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.925949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 11:27:58.928286 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.928268 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 11:27:58.937231 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.937213 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mf675"] Jul 23 11:27:58.949546 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.949521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:58.956499 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-textfile\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.956618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.956756 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.956756 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.956756 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-root\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956756 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956892 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8tc\" (UniqueName: \"kubernetes.io/projected/89e743ad-372a-4439-b672-575b7bfc6f32-kube-api-access-xc8tc\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956892 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-wtmp\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956892 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-metrics-client-ca\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956982 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmt8t\" (UniqueName: \"kubernetes.io/projected/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-api-access-hmt8t\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:58.956982 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956982 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-sys\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:58.956982 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:58.956971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.025275 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.025245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.057911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.057881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-textfile\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.057915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.057949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.058054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.057977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.058054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.058054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.058307 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.058113 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 11:27:59.058307 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.058174 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls podName:e10fae66-1c9b-4bd3-9c54-8259a0e6defd nodeName:}" failed. No retries permitted until 2026-07-23 11:27:59.558154196 +0000 UTC m=+73.540151697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-mf675" (UID: "e10fae66-1c9b-4bd3-9c54-8259a0e6defd") : secret "kube-state-metrics-tls" not found Jul 23 11:27:59.058476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-textfile\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-root\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-root\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8tc\" (UniqueName: \"kubernetes.io/projected/89e743ad-372a-4439-b672-575b7bfc6f32-kube-api-access-xc8tc\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-wtmp\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.058512 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-metrics-client-ca\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.058563 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls podName:89e743ad-372a-4439-b672-575b7bfc6f32 nodeName:}" failed. No retries permitted until 2026-07-23 11:27:59.558545673 +0000 UTC m=+73.540543168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls") pod "node-exporter-8fsl4" (UID: "89e743ad-372a-4439-b672-575b7bfc6f32") : secret "node-exporter-tls" not found Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmt8t\" (UniqueName: \"kubernetes.io/projected/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-api-access-hmt8t\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.058714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-sys\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.059072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.059072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.059072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.058871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.059072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.059001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-sys\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.059072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.059063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-wtmp\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.059310 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.059282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-metrics-client-ca\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.059419 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.059399 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.059551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.059530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.060659 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.060635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.060782 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.060678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.076018 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.075996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmt8t\" (UniqueName: \"kubernetes.io/projected/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-api-access-hmt8t\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.078001 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.077977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8tc\" (UniqueName: \"kubernetes.io/projected/89e743ad-372a-4439-b672-575b7bfc6f32-kube-api-access-xc8tc\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.128893 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.128854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.231255 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.231218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.369357 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.369323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.563351 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.563269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:27:59.563351 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.563308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.563534 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.563421 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 11:27:59.563534 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:27:59.563502 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls podName:e10fae66-1c9b-4bd3-9c54-8259a0e6defd nodeName:}" failed. No retries permitted until 2026-07-23 11:28:00.563479377 +0000 UTC m=+74.545476857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-mf675" (UID: "e10fae66-1c9b-4bd3-9c54-8259a0e6defd") : secret "kube-state-metrics-tls" not found Jul 23 11:27:59.565633 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.565612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e743ad-372a-4439-b672-575b7bfc6f32-node-exporter-tls\") pod \"node-exporter-8fsl4\" (UID: \"89e743ad-372a-4439-b672-575b7bfc6f32\") " pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.736509 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.736473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.831068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.830993 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fsl4" Jul 23 11:27:59.841688 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:27:59.841662 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e743ad_372a_4439_b672_575b7bfc6f32.slice/crio-153703afd8b09d6f341e167f5f3cecfc23dce23ff235904db5537ec4cc5246c4 WatchSource:0}: Error finding container 153703afd8b09d6f341e167f5f3cecfc23dce23ff235904db5537ec4cc5246c4: Status 404 returned error can't find the container with id 153703afd8b09d6f341e167f5f3cecfc23dce23ff235904db5537ec4cc5246c4 Jul 23 11:27:59.856347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.856323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.941362 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.941329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:27:59.946731 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.946705 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:27:59.949727 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.949710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:27:59.952332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 11:27:59.952332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 11:27:59.952459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952315 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 11:27:59.952897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952879 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 11:27:59.952990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952946 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 11:27:59.952990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h9psc\"" Jul 23 11:27:59.952990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952907 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 11:27:59.953158 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 11:27:59.953158 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952946 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 11:27:59.953158 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.952965 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 11:27:59.965133 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.965106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:27:59.998404 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:27:59.998375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fsl4" event={"ID":"89e743ad-372a-4439-b672-575b7bfc6f32","Type":"ContainerStarted","Data":"153703afd8b09d6f341e167f5f3cecfc23dce23ff235904db5537ec4cc5246c4"} Jul 23 11:28:00.067567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.067567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.067567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8vm5\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.068050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.067990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8vm5\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:28:00.169709 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.169880 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.169850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.170208 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.170180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.170260 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:28:00.170234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls podName:23f55182-e8d8-43d7-90fa-1faa1910c328 nodeName:}" failed. No retries permitted until 2026-07-23 11:28:00.670213912 +0000 UTC m=+74.652211393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328") : secret "alertmanager-main-tls" not found Jul 23 11:28:00.171099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.171070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.171431 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.171408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.172557 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.172529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.172657 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.172587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.172724 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.172654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.172900 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.172879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.173010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.172990 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.173497 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.173476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.173939 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.173917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.174262 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.174248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.180229 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.180212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8vm5\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.285688 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.285659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:00.573501 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.573461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:28:00.576350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.576327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e10fae66-1c9b-4bd3-9c54-8259a0e6defd-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mf675\" (UID: \"e10fae66-1c9b-4bd3-9c54-8259a0e6defd\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:28:00.630734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.630686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:00.674673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.674562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.678644 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.678579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.718932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.718900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" Jul 23 11:28:00.729500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.729470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:00.859243 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.859173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:28:00.922576 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.922473 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dd4b8fc4b-75vcl" Jul 23 11:28:00.931964 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:00.931895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mf675"] Jul 23 11:28:00.938726 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:28:00.938693 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10fae66_1c9b_4bd3_9c54_8259a0e6defd.slice/crio-1c8a5aacf142177064e85cb56df86fda70aa590c53b8459447ad5b9b04cb7e4b WatchSource:0}: Error finding container 1c8a5aacf142177064e85cb56df86fda70aa590c53b8459447ad5b9b04cb7e4b: Status 404 returned error can't find the container with id 1c8a5aacf142177064e85cb56df86fda70aa590c53b8459447ad5b9b04cb7e4b Jul 23 11:28:01.003491 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.003460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" event={"ID":"e10fae66-1c9b-4bd3-9c54-8259a0e6defd","Type":"ContainerStarted","Data":"1c8a5aacf142177064e85cb56df86fda70aa590c53b8459447ad5b9b04cb7e4b"} Jul 23 11:28:01.005112 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.005087 2564 generic.go:356] "Generic (PLEG): container finished" podID="89e743ad-372a-4439-b672-575b7bfc6f32" containerID="88d9cff33870828fdc8a5adfa9acd6aecdec158d5d63d2e544636ea886564320" exitCode=0 Jul 23 11:28:01.005206 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.005120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fsl4" event={"ID":"89e743ad-372a-4439-b672-575b7bfc6f32","Type":"ContainerDied","Data":"88d9cff33870828fdc8a5adfa9acd6aecdec158d5d63d2e544636ea886564320"} Jul 23 11:28:01.032304 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.032283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:28:01.036680 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:28:01.036656 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f55182_e8d8_43d7_90fa_1faa1910c328.slice/crio-303b6ef803b043a1896f5b9341383f5c96fdcd80333adad9e3f2929c1fd22a68 WatchSource:0}: Error finding container 303b6ef803b043a1896f5b9341383f5c96fdcd80333adad9e3f2929c1fd22a68: Status 404 returned error can't find the container with id 303b6ef803b043a1896f5b9341383f5c96fdcd80333adad9e3f2929c1fd22a68 Jul 23 11:28:01.051171 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.051145 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc"] Jul 23 11:28:01.053544 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.053529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.058938 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.058859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 11:28:01.058938 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.058903 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lkjlc\"" Jul 23 11:28:01.058938 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.058903 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 11:28:01.059151 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.059139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f50n2k5jjn8pc\"" Jul 23 11:28:01.059425 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.059408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 11:28:01.059696 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.059673 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 11:28:01.059696 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.059691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 11:28:01.083221 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.083175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc"] Jul 23 11:28:01.175949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.175919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:01.179852 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.179977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.179977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-grpc-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.179977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntlk\" (UniqueName: \"kubernetes.io/projected/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-kube-api-access-9ntlk\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.179977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.180154 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.179987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.180154 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.180028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.180154 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.180096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-metrics-client-ca\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.280962 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.280930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.280970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-grpc-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntlk\" (UniqueName: \"kubernetes.io/projected/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-kube-api-access-9ntlk\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281371 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281371 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.281371 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.281294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-metrics-client-ca\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.282250 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.282170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:01.282467 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.282300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-metrics-client-ca\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284137 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-grpc-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284137 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284325 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-tls\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284436 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.284965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.284949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.292068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.292049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntlk\" (UniqueName: \"kubernetes.io/projected/1a0f6faa-b0a6-4a57-834b-2b55aaae402f-kube-api-access-9ntlk\") pod \"thanos-querier-6cf88f8b97-ks9jc\" (UID: \"1a0f6faa-b0a6-4a57-834b-2b55aaae402f\") " pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.374447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.374356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:01.473651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.473618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:01.506871 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.506842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc"] Jul 23 11:28:01.509949 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:28:01.509924 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0f6faa_b0a6_4a57_834b_2b55aaae402f.slice/crio-e314a40f20705c1387568c26ef4574b39e9642a5937e4f04a58d4a84d894cc22 WatchSource:0}: Error finding container e314a40f20705c1387568c26ef4574b39e9642a5937e4f04a58d4a84d894cc22: Status 404 returned error can't find the container with id e314a40f20705c1387568c26ef4574b39e9642a5937e4f04a58d4a84d894cc22 Jul 23 11:28:01.579868 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.579833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:01.674210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.674123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:01.947450 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:01.947367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:02.013839 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.013805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fsl4" event={"ID":"89e743ad-372a-4439-b672-575b7bfc6f32","Type":"ContainerStarted","Data":"4c495a43778c72dbce628bde421ea37d989417b3440b5b53d436cbaa9c721477"} Jul 23 11:28:02.014018 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.013870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fsl4" event={"ID":"89e743ad-372a-4439-b672-575b7bfc6f32","Type":"ContainerStarted","Data":"1409329a0a6f03caf52fed9ea038103a04a606d3b1f875815212a88ad45a861a"} Jul 23 11:28:02.016051 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.016018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"e314a40f20705c1387568c26ef4574b39e9642a5937e4f04a58d4a84d894cc22"} Jul 23 11:28:02.017551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.017521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"303b6ef803b043a1896f5b9341383f5c96fdcd80333adad9e3f2929c1fd22a68"} Jul 23 11:28:02.042508 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.042442 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8fsl4" podStartSLOduration=2.989872524 podStartE2EDuration="4.042421723s" totalImagesPullingTime="969.241791ms" totalInitContainerRuntime="83.307408ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:02.040989037 +0000 UTC m=+76.022986542" watchObservedRunningTime="2026-07-23 11:28:02.042421723 +0000 UTC m=+76.024419231" Jul 23 11:28:02.051048 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.051006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:02.154466 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.154406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:02.261276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:02.261246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:03.023872 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.023836 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590" exitCode=0 Jul 23 11:28:03.024072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.023973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590"} Jul 23 11:28:03.026645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.026514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" event={"ID":"e10fae66-1c9b-4bd3-9c54-8259a0e6defd","Type":"ContainerStarted","Data":"1d7df380e4b77c7bdfc5fce6e2f1afdc2ebe7fa537730e1a1c95af895dd0ff13"} Jul 23 11:28:03.026645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.026573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" event={"ID":"e10fae66-1c9b-4bd3-9c54-8259a0e6defd","Type":"ContainerStarted","Data":"0b6af225e1e3f9e404f0b66cd5fb8b7a30529011e9d7d189afaa0fc38088e226"} Jul 23 11:28:03.026645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.026602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" event={"ID":"e10fae66-1c9b-4bd3-9c54-8259a0e6defd","Type":"ContainerStarted","Data":"7826320d09f4b7203a99906260094ef7febe6753225d67c75fa177fecde02afb"} Jul 23 11:28:03.100942 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.100882 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-mf675" podStartSLOduration=3.538830485 podStartE2EDuration="5.100862669s" totalImagesPullingTime="1.562032184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:03.098394616 +0000 UTC m=+77.080392119" watchObservedRunningTime="2026-07-23 11:28:03.100862669 +0000 UTC m=+77.082860173" Jul 23 11:28:03.422336 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.422230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:03.519075 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.519044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:03.698087 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.697957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:03.756706 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.755497 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:03.757839 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.757815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.762852 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.762829 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 11:28:03.763196 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.763174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 11:28:03.763459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.763443 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 11:28:03.763567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.763451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2dh7r\"" Jul 23 11:28:03.764359 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.764340 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 11:28:03.764461 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.764367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 11:28:03.764461 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.764367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 11:28:03.768932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.768905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 11:28:03.771878 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.771857 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 11:28:03.787709 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.787679 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:03.826554 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.826516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:03.903496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.903947 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.903753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55srh\" (UniqueName: \"kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:03.984911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:03.984876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:04.004910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.004877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55srh\" (UniqueName: \"kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.004939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.004963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.004991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.005019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.005048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.005080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.005925 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.005897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.006081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.005925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.006476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.006425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.012828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.012799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.015242 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.015195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.019449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.019424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55srh\" (UniqueName: \"kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.019913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.019891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle\") pod \"console-787565797d-2mf6c\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.071378 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.071345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:04.215046 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.215007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:04.457979 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:28:04.457952 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85ad23c_13b8_49e1_a6cc_bbd3a7fdd308.slice/crio-d8fc80e798ad5a6c508736c775aaf063751938e5ee6119c2caa4a78605b382d3 WatchSource:0}: Error finding container d8fc80e798ad5a6c508736c775aaf063751938e5ee6119c2caa4a78605b382d3: Status 404 returned error can't find the container with id d8fc80e798ad5a6c508736c775aaf063751938e5ee6119c2caa4a78605b382d3 Jul 23 11:28:04.934288 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:04.934255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:05.020940 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.019385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:05.036348 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.036307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"8201741c3c0bbe1134d0c2eea7f2c2d3384c9e9dcca727f86a479e11a8e30fe3"} Jul 23 11:28:05.036528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.036380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"ef6f2add4323e177879c83f529902f2fd15b30b15aee54d4d0c48b72fd182601"} Jul 23 11:28:05.036528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.036415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"e9c760ac944cb06f192a698b8578b7842e429e52cfe7450914c1ce3352b2f2c7"} Jul 23 11:28:05.039180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.039158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf"} Jul 23 11:28:05.039282 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.039222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9"} Jul 23 11:28:05.039282 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.039251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826"} Jul 23 11:28:05.039365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.039297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190"} Jul 23 11:28:05.039365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.039322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe"} Jul 23 11:28:05.040308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.040286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787565797d-2mf6c" event={"ID":"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308","Type":"ContainerStarted","Data":"d8fc80e798ad5a6c508736c775aaf063751938e5ee6119c2caa4a78605b382d3"} Jul 23 11:28:05.260724 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.260685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:05.535041 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:05.534979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:06.048347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.048251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"6f23d76b2cddf65af20bdc48ade8bc64d0f5270ad60de50aa141d04da106e06f"} Jul 23 11:28:06.048347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.048336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"ddf11b05ce71be3771f8e911799994ef81d1fc3f8715284d9ea7c6dfcf1ae22e"} Jul 23 11:28:06.048567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.048377 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" event={"ID":"1a0f6faa-b0a6-4a57-834b-2b55aaae402f","Type":"ContainerStarted","Data":"1a9db56edf7517df32dda442e1a8e899ad9afb874f220f8f9e293d428c21cf20"} Jul 23 11:28:06.048567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.048538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:06.052561 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.052533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerStarted","Data":"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29"} Jul 23 11:28:06.074738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.074686 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" podStartSLOduration=2.488450366 podStartE2EDuration="6.074667145s" totalImagesPullingTime="3.586216779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:06.070750541 +0000 UTC m=+80.052748054" watchObservedRunningTime="2026-07-23 11:28:06.074667145 +0000 UTC m=+80.056664647" Jul 23 11:28:06.107731 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.107672 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.287448677 podStartE2EDuration="7.107653878s" totalImagesPullingTime="3.795077716s" totalInitContainerRuntime="25.127485ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:27:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:06.105666654 +0000 UTC m=+80.087664168" watchObservedRunningTime="2026-07-23 11:28:06.107653878 +0000 UTC m=+80.089651381" Jul 23 11:28:06.271325 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.271286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:06.371286 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:06.371155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:08.063532 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:08.063502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787565797d-2mf6c" event={"ID":"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308","Type":"ContainerStarted","Data":"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff"} Jul 23 11:28:08.085167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:08.085118 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-787565797d-2mf6c" podStartSLOduration=2.000231309 podStartE2EDuration="5.085103857s" totalImagesPullingTime="3.084872548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:08.083452486 +0000 UTC m=+82.065449990" watchObservedRunningTime="2026-07-23 11:28:08.085103857 +0000 UTC m=+82.067101361" Jul 23 11:28:09.777346 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.777308 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:09.782188 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.782162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:09.807962 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.807934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:28:09.810645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.810628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.823797 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.823776 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:28:09.854010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.853979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:09.955271 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm59g\" (UniqueName: \"kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955401 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955535 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:09.955535 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:09.955440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056231 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056231 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm59g\" (UniqueName: \"kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056424 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056424 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056424 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.056571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.056482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.057106 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.057082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.057442 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.057176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.057442 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.057196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.057574 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.057196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.059071 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.059049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.059151 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.059085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.064413 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.064391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm59g\" (UniqueName: \"kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g\") pod \"console-6b87849898-znhgj\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.119457 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.119427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:10.252117 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:10.252092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:28:10.254438 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:28:10.254408 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fb4638_c01b_418d_befd_bb90d5df934f.slice/crio-7100063d70ee3c57e94830f6aa692258f1635b3eceaf1ada0f3883f343ac9eb2 WatchSource:0}: Error finding container 7100063d70ee3c57e94830f6aa692258f1635b3eceaf1ada0f3883f343ac9eb2: Status 404 returned error can't find the container with id 7100063d70ee3c57e94830f6aa692258f1635b3eceaf1ada0f3883f343ac9eb2 Jul 23 11:28:11.077199 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:11.077160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b87849898-znhgj" event={"ID":"d0fb4638-c01b-418d-befd-bb90d5df934f","Type":"ContainerStarted","Data":"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950"} Jul 23 11:28:11.077566 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:11.077216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b87849898-znhgj" event={"ID":"d0fb4638-c01b-418d-befd-bb90d5df934f","Type":"ContainerStarted","Data":"7100063d70ee3c57e94830f6aa692258f1635b3eceaf1ada0f3883f343ac9eb2"} Jul 23 11:28:11.097481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:11.097430 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b87849898-znhgj" podStartSLOduration=2.097414558 podStartE2EDuration="2.097414558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:28:11.096251571 +0000 UTC m=+85.078249075" watchObservedRunningTime="2026-07-23 11:28:11.097414558 +0000 UTC m=+85.079412060" Jul 23 11:28:12.064412 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:12.064384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cf88f8b97-ks9jc" Jul 23 11:28:12.540968 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:12.540938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:12.626630 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:12.626591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:14.072232 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:14.072191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:15.617918 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:15.617884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:15.674309 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:15.674276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:20.120006 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:20.119969 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:20.120377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:20.120070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:20.124814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:20.124794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:21.118240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:21.118215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:28:21.311484 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:21.311452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:21.412063 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:21.411973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:27.650048 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:27.650009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:28.994534 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:28.994497 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2dfxr" Jul 23 11:28:30.557279 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:30.557244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:30.660519 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:30.660479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:30.721059 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:30.721028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:34.618272 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:34.618234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:35.092528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.092494 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-787565797d-2mf6c" podUID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" containerName="console" containerID="cri-o://45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff" gracePeriod=15 Jul 23 11:28:35.324170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.324149 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:35.371148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371114 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371175 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371243 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55srh\" (UniqueName: \"kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371340 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca\") pod \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\" (UID: \"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308\") " Jul 23 11:28:35.371558 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:35.371627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371578 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:35.371689 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371641 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-trusted-ca-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.371746 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371686 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config" (OuterVolumeSpecName: "console-config") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:35.372025 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.371889 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca" (OuterVolumeSpecName: "service-ca") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:28:35.373459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.373433 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:35.373459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.373445 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:28:35.373567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.373530 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh" (OuterVolumeSpecName: "kube-api-access-55srh") pod "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" (UID: "a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308"). InnerVolumeSpecName "kube-api-access-55srh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:28:35.472851 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472817 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.472851 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472844 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-oauth-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.472851 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472853 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-oauth-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.472851 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472862 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-console-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.473096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472869 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55srh\" (UniqueName: \"kubernetes.io/projected/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-kube-api-access-55srh\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:35.473096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:35.472878 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:28:36.168282 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.168241 2564 generic.go:356] "Generic (PLEG): container finished" podID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" containerID="45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff" exitCode=0 Jul 23 11:28:36.168749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.168290 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787565797d-2mf6c" Jul 23 11:28:36.168749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.168333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787565797d-2mf6c" event={"ID":"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308","Type":"ContainerDied","Data":"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff"} Jul 23 11:28:36.168749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.168398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787565797d-2mf6c" event={"ID":"a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308","Type":"ContainerDied","Data":"d8fc80e798ad5a6c508736c775aaf063751938e5ee6119c2caa4a78605b382d3"} Jul 23 11:28:36.168749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.168430 2564 scope.go:117] "RemoveContainer" containerID="45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff" Jul 23 11:28:36.176964 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.176947 2564 scope.go:117] "RemoveContainer" containerID="45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff" Jul 23 11:28:36.177207 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:28:36.177187 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff\": container with ID starting with 45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff not found: ID does not exist" containerID="45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff" Jul 23 11:28:36.177251 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.177217 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff"} err="failed to get container status \"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff\": rpc error: code = NotFound desc = could not find container \"45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff\": container with ID starting with 45ded18b589292138e6c4bc151e8890d2cc0c7de9c4f4ab73ca701b9067d8cff not found: ID does not exist" Jul 23 11:28:36.188942 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.188909 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:36.192810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.192790 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-787565797d-2mf6c"] Jul 23 11:28:36.541025 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:36.540991 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" path="/var/lib/kubelet/pods/a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308/volumes" Jul 23 11:28:38.617537 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:38.617500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:38.714492 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:38.714459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:28:38.814528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:28:38.814498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:15.086292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:15.086262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:15.144700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:15.144665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:19.365295 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365257 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365540 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="alertmanager" containerID="cri-o://5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" gracePeriod=120 Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365566 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy" containerID="cri-o://0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" gracePeriod=120 Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365566 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="prom-label-proxy" containerID="cri-o://d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" gracePeriod=120 Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365603 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-metric" containerID="cri-o://7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" gracePeriod=120 Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365615 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-web" containerID="cri-o://f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" gracePeriod=120 Jul 23 11:29:19.365736 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.365633 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="config-reloader" containerID="cri-o://e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" gracePeriod=120 Jul 23 11:29:19.957187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.957152 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:29:19.957468 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.957456 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" containerName="console" Jul 23 11:29:19.957514 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.957470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" containerName="console" Jul 23 11:29:19.957551 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.957519 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85ad23c-13b8-49e1-a6cc-bbd3a7fdd308" containerName="console" Jul 23 11:29:19.959332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.959315 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:19.971805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:19.971778 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:29:20.021867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.021832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:20.077690 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.077665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:20.112188 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.112438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.112332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpfv4\" (UniqueName: \"kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpfv4\" (UniqueName: \"kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.213679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.213441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.214180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.214159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.214256 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.214179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.214335 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.214251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.214624 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.214601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.216015 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.215991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.216222 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.216201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.222683 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.222662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpfv4\" (UniqueName: \"kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4\") pod \"console-65885c47b6-hxxr4\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.268524 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.268489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:20.344695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344662 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" exitCode=0 Jul 23 11:29:20.344695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344687 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" exitCode=0 Jul 23 11:29:20.344695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344693 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" exitCode=0 Jul 23 11:29:20.344695 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344698 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" exitCode=0 Jul 23 11:29:20.344954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29"} Jul 23 11:29:20.344954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9"} Jul 23 11:29:20.344954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190"} Jul 23 11:29:20.344954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.344898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe"} Jul 23 11:29:20.396734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.396708 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:29:20.399188 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:29:20.399160 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d17cf9_c9a7_4c42_a891_a8604b29e8c0.slice/crio-23f6a2beef5865eb477f58298c9929016e9e3cbdfaa17d2e993281d8cd3fb68a WatchSource:0}: Error finding container 23f6a2beef5865eb477f58298c9929016e9e3cbdfaa17d2e993281d8cd3fb68a: Status 404 returned error can't find the container with id 23f6a2beef5865eb477f58298c9929016e9e3cbdfaa17d2e993281d8cd3fb68a Jul 23 11:29:20.609030 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.609003 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:20.717316 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717278 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8vm5\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717390 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717474 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.717516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") pod \"23f55182-e8d8-43d7-90fa-1faa1910c328\" (UID: \"23f55182-e8d8-43d7-90fa-1faa1910c328\") " Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.717896 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:20.718283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.718059 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.719348 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.719314 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:29:20.719707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.719680 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:20.720918 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.720880 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5" (OuterVolumeSpecName: "kube-api-access-w8vm5") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "kube-api-access-w8vm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:29:20.721245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.721218 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.721328 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.721243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out" (OuterVolumeSpecName: "config-out") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:29:20.721422 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.721395 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:29:20.721555 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.721481 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.722059 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.722032 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume" (OuterVolumeSpecName: "config-volume") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.722338 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.722311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.722531 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.722512 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.724951 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.724928 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.731178 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.731151 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config" (OuterVolumeSpecName: "web-config") pod "23f55182-e8d8-43d7-90fa-1faa1910c328" (UID: "23f55182-e8d8-43d7-90fa-1faa1910c328"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:20.819067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819034 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-web-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819063 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819073 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8vm5\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-kube-api-access-w8vm5\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819083 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-config-out\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819092 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819100 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-config-volume\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819108 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819118 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/23f55182-e8d8-43d7-90fa-1faa1910c328-alertmanager-main-db\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819126 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-cluster-tls-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819144 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23f55182-e8d8-43d7-90fa-1faa1910c328-tls-assets\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819152 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f55182-e8d8-43d7-90fa-1faa1910c328-metrics-client-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:20.819278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:20.819162 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/23f55182-e8d8-43d7-90fa-1faa1910c328-secret-alertmanager-main-tls\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:21.351210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351179 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" exitCode=0 Jul 23 11:29:21.351210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351203 2564 generic.go:356] "Generic (PLEG): container finished" podID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerID="f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" exitCode=0 Jul 23 11:29:21.351457 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351273 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.351457 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf"} Jul 23 11:29:21.351457 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826"} Jul 23 11:29:21.351585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"23f55182-e8d8-43d7-90fa-1faa1910c328","Type":"ContainerDied","Data":"303b6ef803b043a1896f5b9341383f5c96fdcd80333adad9e3f2929c1fd22a68"} Jul 23 11:29:21.351585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.351524 2564 scope.go:117] "RemoveContainer" containerID="d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" Jul 23 11:29:21.353273 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.353249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65885c47b6-hxxr4" event={"ID":"54d17cf9-c9a7-4c42-a891-a8604b29e8c0","Type":"ContainerStarted","Data":"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920"} Jul 23 11:29:21.353369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.353300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65885c47b6-hxxr4" event={"ID":"54d17cf9-c9a7-4c42-a891-a8604b29e8c0","Type":"ContainerStarted","Data":"23f6a2beef5865eb477f58298c9929016e9e3cbdfaa17d2e993281d8cd3fb68a"} Jul 23 11:29:21.359626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.359598 2564 scope.go:117] "RemoveContainer" containerID="7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" Jul 23 11:29:21.366483 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.366466 2564 scope.go:117] "RemoveContainer" containerID="0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" Jul 23 11:29:21.373142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.373123 2564 scope.go:117] "RemoveContainer" containerID="f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" Jul 23 11:29:21.380954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.380918 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65885c47b6-hxxr4" podStartSLOduration=2.3809026859999998 podStartE2EDuration="2.380902686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:29:21.380177081 +0000 UTC m=+155.362174589" watchObservedRunningTime="2026-07-23 11:29:21.380902686 +0000 UTC m=+155.362900191" Jul 23 11:29:21.388462 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.388446 2564 scope.go:117] "RemoveContainer" containerID="e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" Jul 23 11:29:21.395718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.395616 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:21.395856 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.395743 2564 scope.go:117] "RemoveContainer" containerID="5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" Jul 23 11:29:21.402433 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.402400 2564 scope.go:117] "RemoveContainer" containerID="4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590" Jul 23 11:29:21.404718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.404699 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:21.409672 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.409656 2564 scope.go:117] "RemoveContainer" containerID="d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" Jul 23 11:29:21.409940 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.409921 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29\": container with ID starting with d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29 not found: ID does not exist" containerID="d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" Jul 23 11:29:21.409994 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.409949 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29"} err="failed to get container status \"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29\": rpc error: code = NotFound desc = could not find container \"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29\": container with ID starting with d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29 not found: ID does not exist" Jul 23 11:29:21.409994 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.409968 2564 scope.go:117] "RemoveContainer" containerID="7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" Jul 23 11:29:21.410187 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.410172 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf\": container with ID starting with 7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf not found: ID does not exist" containerID="7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" Jul 23 11:29:21.410232 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410190 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf"} err="failed to get container status \"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf\": rpc error: code = NotFound desc = could not find container \"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf\": container with ID starting with 7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf not found: ID does not exist" Jul 23 11:29:21.410232 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410202 2564 scope.go:117] "RemoveContainer" containerID="0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" Jul 23 11:29:21.410392 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.410375 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9\": container with ID starting with 0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9 not found: ID does not exist" containerID="0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" Jul 23 11:29:21.410429 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410395 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9"} err="failed to get container status \"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9\": rpc error: code = NotFound desc = could not find container \"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9\": container with ID starting with 0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9 not found: ID does not exist" Jul 23 11:29:21.410429 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410406 2564 scope.go:117] "RemoveContainer" containerID="f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" Jul 23 11:29:21.410585 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.410571 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826\": container with ID starting with f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826 not found: ID does not exist" containerID="f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" Jul 23 11:29:21.410624 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410587 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826"} err="failed to get container status \"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826\": rpc error: code = NotFound desc = could not find container \"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826\": container with ID starting with f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826 not found: ID does not exist" Jul 23 11:29:21.410624 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410600 2564 scope.go:117] "RemoveContainer" containerID="e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" Jul 23 11:29:21.410795 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.410780 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190\": container with ID starting with e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190 not found: ID does not exist" containerID="e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" Jul 23 11:29:21.410848 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410796 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190"} err="failed to get container status \"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190\": rpc error: code = NotFound desc = could not find container \"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190\": container with ID starting with e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190 not found: ID does not exist" Jul 23 11:29:21.410848 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.410807 2564 scope.go:117] "RemoveContainer" containerID="5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" Jul 23 11:29:21.411002 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.410987 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe\": container with ID starting with 5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe not found: ID does not exist" containerID="5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" Jul 23 11:29:21.411052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411005 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe"} err="failed to get container status \"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe\": rpc error: code = NotFound desc = could not find container \"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe\": container with ID starting with 5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe not found: ID does not exist" Jul 23 11:29:21.411052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411016 2564 scope.go:117] "RemoveContainer" containerID="4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590" Jul 23 11:29:21.411239 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:21.411220 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590\": container with ID starting with 4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590 not found: ID does not exist" containerID="4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590" Jul 23 11:29:21.411276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411247 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590"} err="failed to get container status \"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590\": rpc error: code = NotFound desc = could not find container \"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590\": container with ID starting with 4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590 not found: ID does not exist" Jul 23 11:29:21.411276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411261 2564 scope.go:117] "RemoveContainer" containerID="d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29" Jul 23 11:29:21.411485 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411465 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29"} err="failed to get container status \"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29\": rpc error: code = NotFound desc = could not find container \"d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29\": container with ID starting with d2b309e219d9cadd7a4494c371e9f87c4f16af1f9fa38c97b5c602449faa5f29 not found: ID does not exist" Jul 23 11:29:21.411528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411485 2564 scope.go:117] "RemoveContainer" containerID="7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf" Jul 23 11:29:21.411703 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411680 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf"} err="failed to get container status \"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf\": rpc error: code = NotFound desc = could not find container \"7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf\": container with ID starting with 7a57e2f92a2319c4f41b26d89c8642928a2e2bca0393e2bb278c5e8088b7e8cf not found: ID does not exist" Jul 23 11:29:21.411703 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411702 2564 scope.go:117] "RemoveContainer" containerID="0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9" Jul 23 11:29:21.412010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.411993 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9"} err="failed to get container status \"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9\": rpc error: code = NotFound desc = could not find container \"0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9\": container with ID starting with 0739addec8092808d10931fa9d2b2bbff60a1b3c630e639f09f8b037288f0ba9 not found: ID does not exist" Jul 23 11:29:21.412057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412011 2564 scope.go:117] "RemoveContainer" containerID="f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826" Jul 23 11:29:21.412234 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412217 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826"} err="failed to get container status \"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826\": rpc error: code = NotFound desc = could not find container \"f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826\": container with ID starting with f59f05975ce000e0ce969ad02b111fd8c65ab38afa6b2b4ebcb12f368dc70826 not found: ID does not exist" Jul 23 11:29:21.412272 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412234 2564 scope.go:117] "RemoveContainer" containerID="e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190" Jul 23 11:29:21.412433 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412414 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190"} err="failed to get container status \"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190\": rpc error: code = NotFound desc = could not find container \"e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190\": container with ID starting with e9c149476ca1c34b283a4a40581bf7751287b37353db406fe873a0b4d32d5190 not found: ID does not exist" Jul 23 11:29:21.412484 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412434 2564 scope.go:117] "RemoveContainer" containerID="5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe" Jul 23 11:29:21.412667 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412648 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe"} err="failed to get container status \"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe\": rpc error: code = NotFound desc = could not find container \"5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe\": container with ID starting with 5cd1fe20c8b691d10920689059692ff7ec107228efe8c932bca54a4723e2abbe not found: ID does not exist" Jul 23 11:29:21.412715 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412668 2564 scope.go:117] "RemoveContainer" containerID="4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590" Jul 23 11:29:21.412915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.412892 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590"} err="failed to get container status \"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590\": rpc error: code = NotFound desc = could not find container \"4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590\": container with ID starting with 4afd221bc1f58268bf35a26669acb83533c5ab7d6f5a98d20f83a75d7bf53590 not found: ID does not exist" Jul 23 11:29:21.434263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:21.434507 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434496 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="init-config-reloader" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434511 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="init-config-reloader" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434526 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-metric" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434532 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-metric" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434538 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-web" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434544 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-web" Jul 23 11:29:21.434552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434549 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="alertmanager" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434556 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="alertmanager" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434562 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434567 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434573 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="prom-label-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434578 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="prom-label-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434588 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="config-reloader" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434593 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="config-reloader" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434633 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434640 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-web" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434646 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="prom-label-proxy" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434653 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="alertmanager" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434658 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="config-reloader" Jul 23 11:29:21.434718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.434663 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" containerName="kube-rbac-proxy-metric" Jul 23 11:29:21.437828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.437809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.440339 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 11:29:21.440410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 11:29:21.440410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440369 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 11:29:21.440410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 11:29:21.440725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h9psc\"" Jul 23 11:29:21.440725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 11:29:21.440873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440730 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 11:29:21.440873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 11:29:21.440873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.440795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 11:29:21.446157 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.446139 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 11:29:21.451686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.451666 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:21.524538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-web-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524693 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524693 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524693 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pcs\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-kube-api-access-w8pcs\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524693 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.524974 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.524940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-config-out\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.525018 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.525004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.525055 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.525030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.525088 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.525059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.525126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.525093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.625985 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.625918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.625985 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.625979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626118 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626197 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-config-out\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626344 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626484 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.626484 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.626453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-web-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627398 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627398 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627398 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627632 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pcs\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-kube-api-access-w8pcs\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627632 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.627632 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.627559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4b88c7-03ba-409d-a796-ed7d85070461-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.629164 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.629061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f4b88c7-03ba-409d-a796-ed7d85070461-config-out\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.629743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.629635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.629743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.629682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.629971 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.629949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-web-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.630054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.630038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.630137 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.630120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.630367 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.630351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.630455 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.630438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.631134 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.631116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f4b88c7-03ba-409d-a796-ed7d85070461-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.638458 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.638435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pcs\" (UniqueName: \"kubernetes.io/projected/9f4b88c7-03ba-409d-a796-ed7d85070461-kube-api-access-w8pcs\") pod \"alertmanager-main-0\" (UID: \"9f4b88c7-03ba-409d-a796-ed7d85070461\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.747204 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.747171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:29:21.879674 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:21.879647 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:29:21.882314 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:29:21.882286 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4b88c7_03ba_409d_a796_ed7d85070461.slice/crio-f42393a94676a1f440c88d944232edce305b58484ecdefc590cee203ffae705f WatchSource:0}: Error finding container f42393a94676a1f440c88d944232edce305b58484ecdefc590cee203ffae705f: Status 404 returned error can't find the container with id f42393a94676a1f440c88d944232edce305b58484ecdefc590cee203ffae705f Jul 23 11:29:22.359712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:22.359680 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f4b88c7-03ba-409d-a796-ed7d85070461" containerID="635f07ba0c0acb855fc0eb3c8c72579b91d6339d781e07f48ad1cfa004633a45" exitCode=0 Jul 23 11:29:22.359902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:22.359796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerDied","Data":"635f07ba0c0acb855fc0eb3c8c72579b91d6339d781e07f48ad1cfa004633a45"} Jul 23 11:29:22.359902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:22.359858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"f42393a94676a1f440c88d944232edce305b58484ecdefc590cee203ffae705f"} Jul 23 11:29:22.541958 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:22.541935 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f55182-e8d8-43d7-90fa-1faa1910c328" path="/var/lib/kubelet/pods/23f55182-e8d8-43d7-90fa-1faa1910c328/volumes" Jul 23 11:29:23.366905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.366870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"a93dc040070771ffb013a07a74630b898e07bcfe365c461d5e847aa3b91a78fa"} Jul 23 11:29:23.367099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.366946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"7a7fe655233e72a6b6ce426a0f5e698cde632b6ed36844ea969c96f77301b5aa"} Jul 23 11:29:23.367099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.366981 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"832d725afde7c7e48788d327f47f4ccf232ab27a425cd808bc9255f3412f54a2"} Jul 23 11:29:23.367099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.367017 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"72950d64f89211d8fda5e4a4b35b28f1d383f2af1b75f110885d1ed0a62bfbce"} Jul 23 11:29:23.367099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.367052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"af86b6ea520681c53e7a805b1061eb62e8d750aee70e816f4f95667512203982"} Jul 23 11:29:23.367099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.367085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f4b88c7-03ba-409d-a796-ed7d85070461","Type":"ContainerStarted","Data":"f1493635733015bd8366b6d7cc72414c1a87bb1311f521ea169d8ae63c2ac3c2"} Jul 23 11:29:23.392686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:23.392628 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.377363477 podStartE2EDuration="2.392607138s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.243661ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:29:23.391747394 +0000 UTC m=+157.373744898" watchObservedRunningTime="2026-07-23 11:29:23.392607138 +0000 UTC m=+157.374604642" Jul 23 11:29:26.996746 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:26.996716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:27.096936 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:27.096898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:30.269303 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.269268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:30.269787 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.269331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:30.274493 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.274468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:30.399176 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.399147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:29:30.480137 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.480110 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:29:30.597026 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.596935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:30.695136 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.695110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:30.794554 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.794522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:30.866977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:30.866902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:48.663811 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:48.663747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:48.759960 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:48.759923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:49.309013 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.308976 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk"] Jul 23 11:29:49.311915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.311898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.314585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.314563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jzk2n\"" Jul 23 11:29:49.314669 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.314573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:29:49.314744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.314597 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:29:49.321424 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.321405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk"] Jul 23 11:29:49.348129 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.348100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.348129 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.348131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.348330 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.348172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5k6\" (UniqueName: \"kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.449652 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.449607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.449652 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.449647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.449902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.449684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5k6\" (UniqueName: \"kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.450047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.450010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.450080 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.450047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.457969 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.457943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5k6\" (UniqueName: \"kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.621347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.621262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:29:49.761581 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:49.761557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk"] Jul 23 11:29:49.764088 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:29:49.764064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795e7ff0_e82b_4902_8506_98d9809e8572.slice/crio-f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1 WatchSource:0}: Error finding container f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1: Status 404 returned error can't find the container with id f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1 Jul 23 11:29:50.281217 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:50.281186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:29:50.470447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:50.470407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" event={"ID":"795e7ff0-e82b-4902-8506-98d9809e8572","Type":"ContainerStarted","Data":"f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1"} Jul 23 11:29:55.500881 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:55.500835 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b87849898-znhgj" podUID="d0fb4638-c01b-418d-befd-bb90d5df934f" containerName="console" containerID="cri-o://59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950" gracePeriod=15 Jul 23 11:29:56.371358 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.371333 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:29:56.413859 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.413822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.413880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.413914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm59g\" (UniqueName: \"kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.413949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.413979 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414287 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config\") pod \"d0fb4638-c01b-418d-befd-bb90d5df934f\" (UID: \"d0fb4638-c01b-418d-befd-bb90d5df934f\") " Jul 23 11:29:56.414462 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414434 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:56.414528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414510 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:56.414699 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414535 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca" (OuterVolumeSpecName: "service-ca") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:56.414699 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.414636 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config" (OuterVolumeSpecName: "console-config") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:29:56.416244 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.416217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:56.416343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.416251 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:29:56.416493 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.416476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g" (OuterVolumeSpecName: "kube-api-access-wm59g") pod "d0fb4638-c01b-418d-befd-bb90d5df934f" (UID: "d0fb4638-c01b-418d-befd-bb90d5df934f"). InnerVolumeSpecName "kube-api-access-wm59g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:29:56.496175 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.496079 2564 generic.go:356] "Generic (PLEG): container finished" podID="d0fb4638-c01b-418d-befd-bb90d5df934f" containerID="59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950" exitCode=0 Jul 23 11:29:56.496175 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.496131 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b87849898-znhgj" Jul 23 11:29:56.496408 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.496181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b87849898-znhgj" event={"ID":"d0fb4638-c01b-418d-befd-bb90d5df934f","Type":"ContainerDied","Data":"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950"} Jul 23 11:29:56.496408 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.496238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b87849898-znhgj" event={"ID":"d0fb4638-c01b-418d-befd-bb90d5df934f","Type":"ContainerDied","Data":"7100063d70ee3c57e94830f6aa692258f1635b3eceaf1ada0f3883f343ac9eb2"} Jul 23 11:29:56.496408 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.496265 2564 scope.go:117] "RemoveContainer" containerID="59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950" Jul 23 11:29:56.497918 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.497892 2564 generic.go:356] "Generic (PLEG): container finished" podID="795e7ff0-e82b-4902-8506-98d9809e8572" containerID="56773290cc03ca7d5104b5367e277c1390286935ed9bbca5fd662d3d7f3e94f7" exitCode=0 Jul 23 11:29:56.498028 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.497973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" event={"ID":"795e7ff0-e82b-4902-8506-98d9809e8572","Type":"ContainerDied","Data":"56773290cc03ca7d5104b5367e277c1390286935ed9bbca5fd662d3d7f3e94f7"} Jul 23 11:29:56.505384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.505218 2564 scope.go:117] "RemoveContainer" containerID="59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950" Jul 23 11:29:56.505613 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:29:56.505495 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950\": container with ID starting with 59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950 not found: ID does not exist" containerID="59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950" Jul 23 11:29:56.505613 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.505519 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950"} err="failed to get container status \"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950\": rpc error: code = NotFound desc = could not find container \"59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950\": container with ID starting with 59c1559370cd2e4cf8bf73adde91a807b564220464deb4c7082b440680c73950 not found: ID does not exist" Jul 23 11:29:56.514896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514874 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-trusted-ca-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.514896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514898 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-oauth-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.515021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514911 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.515021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514922 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-console-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.515021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514935 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0fb4638-c01b-418d-befd-bb90d5df934f-console-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.515021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514948 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0fb4638-c01b-418d-befd-bb90d5df934f-oauth-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.515021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.514961 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm59g\" (UniqueName: \"kubernetes.io/projected/d0fb4638-c01b-418d-befd-bb90d5df934f-kube-api-access-wm59g\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:29:56.531423 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.531397 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:29:56.535109 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.535089 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b87849898-znhgj"] Jul 23 11:29:56.541682 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:56.541653 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fb4638-c01b-418d-befd-bb90d5df934f" path="/var/lib/kubelet/pods/d0fb4638-c01b-418d-befd-bb90d5df934f/volumes" Jul 23 11:29:58.508186 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:58.508151 2564 generic.go:356] "Generic (PLEG): container finished" podID="795e7ff0-e82b-4902-8506-98d9809e8572" containerID="1811826bf9122814ed9906291bf3ebb01a74334f27a6ecc9f46a8f6ff952cad5" exitCode=0 Jul 23 11:29:58.508565 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:29:58.508211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" event={"ID":"795e7ff0-e82b-4902-8506-98d9809e8572","Type":"ContainerDied","Data":"1811826bf9122814ed9906291bf3ebb01a74334f27a6ecc9f46a8f6ff952cad5"} Jul 23 11:30:05.538187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:05.538148 2564 generic.go:356] "Generic (PLEG): container finished" podID="795e7ff0-e82b-4902-8506-98d9809e8572" containerID="cd13fee54a09e14c88465dd82dbaf34bfea80885c7166eeab78427491d784403" exitCode=0 Jul 23 11:30:05.538563 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:05.538226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" event={"ID":"795e7ff0-e82b-4902-8506-98d9809e8572","Type":"ContainerDied","Data":"cd13fee54a09e14c88465dd82dbaf34bfea80885c7166eeab78427491d784403"} Jul 23 11:30:06.669172 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.669147 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:30:06.805823 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.805711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util\") pod \"795e7ff0-e82b-4902-8506-98d9809e8572\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " Jul 23 11:30:06.805823 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.805809 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle\") pod \"795e7ff0-e82b-4902-8506-98d9809e8572\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " Jul 23 11:30:06.806002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.805837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq5k6\" (UniqueName: \"kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6\") pod \"795e7ff0-e82b-4902-8506-98d9809e8572\" (UID: \"795e7ff0-e82b-4902-8506-98d9809e8572\") " Jul 23 11:30:06.806451 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.806416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle" (OuterVolumeSpecName: "bundle") pod "795e7ff0-e82b-4902-8506-98d9809e8572" (UID: "795e7ff0-e82b-4902-8506-98d9809e8572"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:06.808246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.808221 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6" (OuterVolumeSpecName: "kube-api-access-qq5k6") pod "795e7ff0-e82b-4902-8506-98d9809e8572" (UID: "795e7ff0-e82b-4902-8506-98d9809e8572"). InnerVolumeSpecName "kube-api-access-qq5k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:30:06.812863 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.812838 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util" (OuterVolumeSpecName: "util") pod "795e7ff0-e82b-4902-8506-98d9809e8572" (UID: "795e7ff0-e82b-4902-8506-98d9809e8572"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:06.906587 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.906558 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:06.906587 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.906581 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795e7ff0-e82b-4902-8506-98d9809e8572-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:06.906587 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:06.906591 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5k6\" (UniqueName: \"kubernetes.io/projected/795e7ff0-e82b-4902-8506-98d9809e8572-kube-api-access-qq5k6\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:07.547568 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:07.547534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" event={"ID":"795e7ff0-e82b-4902-8506-98d9809e8572","Type":"ContainerDied","Data":"f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1"} Jul 23 11:30:07.547568 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:07.547553 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fmjpk" Jul 23 11:30:07.547795 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:07.547581 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9df98507bc154330d529cd182dec35fdb6c147afc8091d19dfc512a51d6cea1" Jul 23 11:30:09.057724 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:09.057694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:09.114644 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:09.114611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:10.150300 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:10.150266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:11.498027 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:11.497991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:12.263296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263264 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc"] Jul 23 11:30:12.263571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263558 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fb4638-c01b-418d-befd-bb90d5df934f" containerName="console" Jul 23 11:30:12.263623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263573 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fb4638-c01b-418d-befd-bb90d5df934f" containerName="console" Jul 23 11:30:12.263623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263586 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="extract" Jul 23 11:30:12.263623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263591 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="extract" Jul 23 11:30:12.263623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263600 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="util" Jul 23 11:30:12.263623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263619 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="util" Jul 23 11:30:12.263814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263627 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="pull" Jul 23 11:30:12.263814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263634 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="pull" Jul 23 11:30:12.263814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263677 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fb4638-c01b-418d-befd-bb90d5df934f" containerName="console" Jul 23 11:30:12.263814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.263684 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="795e7ff0-e82b-4902-8506-98d9809e8572" containerName="extract" Jul 23 11:30:12.298042 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.298014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc"] Jul 23 11:30:12.298182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.298102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.301890 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.301868 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 11:30:12.302002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.301910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tjtld\"" Jul 23 11:30:12.302841 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.302819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:30:12.450783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.450738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.450946 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.450803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44b5m\" (UniqueName: \"kubernetes.io/projected/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-kube-api-access-44b5m\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.551930 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.551868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.551930 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.551909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44b5m\" (UniqueName: \"kubernetes.io/projected/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-kube-api-access-44b5m\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.552281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.552212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.563344 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.563315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44b5m\" (UniqueName: \"kubernetes.io/projected/1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c-kube-api-access-44b5m\") pod \"cert-manager-operator-controller-manager-55c444bf86-c55cc\" (UID: \"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.606757 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.606729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" Jul 23 11:30:12.755192 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:12.755087 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc"] Jul 23 11:30:12.758207 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:12.758159 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb5e2e0_9940_46f5_80cb_7a68c43cbb5c.slice/crio-ab89d4febfce1096f3891355e03102bb7c080b301e84fe6c27908f69470f1231 WatchSource:0}: Error finding container ab89d4febfce1096f3891355e03102bb7c080b301e84fe6c27908f69470f1231: Status 404 returned error can't find the container with id ab89d4febfce1096f3891355e03102bb7c080b301e84fe6c27908f69470f1231 Jul 23 11:30:13.569390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:13.569337 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" event={"ID":"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c","Type":"ContainerStarted","Data":"ab89d4febfce1096f3891355e03102bb7c080b301e84fe6c27908f69470f1231"} Jul 23 11:30:13.926325 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:13.926243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:14.027019 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:14.026988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:15.578403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:15.578368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" event={"ID":"1bb5e2e0-9940-46f5-80cb-7a68c43cbb5c","Type":"ContainerStarted","Data":"3a069d5cf515c915ed227116a32ec61d4639c6909a9e4c2c2b8088b5cfa01125"} Jul 23 11:30:15.599575 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:15.599512 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c55cc" podStartSLOduration=1.5168602180000001 podStartE2EDuration="3.599491223s" totalImagesPullingTime="2.082631005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:15.596931355 +0000 UTC m=+209.578928900" watchObservedRunningTime="2026-07-23 11:30:15.599491223 +0000 UTC m=+209.581488728" Jul 23 11:30:15.687384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:15.687339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:15.790162 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:15.790111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:16.035974 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.035934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:16.138078 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.138039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:16.774253 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.774219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj"] Jul 23 11:30:16.797377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.797336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj"] Jul 23 11:30:16.797569 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.797436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.800035 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.800016 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:30:16.800853 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.800838 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jzk2n\"" Jul 23 11:30:16.800911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.800858 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:30:16.887032 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.886998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784xj\" (UniqueName: \"kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.887202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.887054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.887202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.887180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.988193 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.988153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.988365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.988201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784xj\" (UniqueName: \"kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.988365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.988241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.988622 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.988602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:16.988664 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:16.988609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:17.004875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.004845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-784xj\" (UniqueName: \"kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:17.106280 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.106185 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:17.260444 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.260416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj"] Jul 23 11:30:17.262674 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:17.262647 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae3c30_ee70_46f7_b9eb_a16e42c9801f.slice/crio-48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4 WatchSource:0}: Error finding container 48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4: Status 404 returned error can't find the container with id 48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4 Jul 23 11:30:17.587418 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.587380 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerID="ddd11f51c861c7d2c3537f225f2ff40ac4bdc305e171555361c7b051890ebfc5" exitCode=0 Jul 23 11:30:17.587600 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.587479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" event={"ID":"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f","Type":"ContainerDied","Data":"ddd11f51c861c7d2c3537f225f2ff40ac4bdc305e171555361c7b051890ebfc5"} Jul 23 11:30:17.587600 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.587530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" event={"ID":"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f","Type":"ContainerStarted","Data":"48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4"} Jul 23 11:30:17.988581 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:17.988549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:18.090571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.090538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:18.260419 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.260330 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sblzv"] Jul 23 11:30:18.283126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.283098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sblzv"] Jul 23 11:30:18.283282 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.283192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.285688 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.285660 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 11:30:18.286596 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.286576 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 11:30:18.286704 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.286576 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7kcj2\"" Jul 23 11:30:18.401302 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.401271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.401498 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.401313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnxk\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-kube-api-access-hpnxk\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.502363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.502326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.502534 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.502381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnxk\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-kube-api-access-hpnxk\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.511316 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.511247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.511514 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.511356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnxk\" (UniqueName: \"kubernetes.io/projected/9f842103-20f5-40be-9321-067a1ff53fe2-kube-api-access-hpnxk\") pod \"cert-manager-cainjector-559cdd8c87-sblzv\" (UID: \"9f842103-20f5-40be-9321-067a1ff53fe2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.605458 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.605422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" Jul 23 11:30:18.767327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:18.767253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sblzv"] Jul 23 11:30:18.770988 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:18.770961 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f842103_20f5_40be_9321_067a1ff53fe2.slice/crio-c3fe2f04ee806d1967f19adf797e99f69ddf7416ac3945d2548178395d2f89d8 WatchSource:0}: Error finding container c3fe2f04ee806d1967f19adf797e99f69ddf7416ac3945d2548178395d2f89d8: Status 404 returned error can't find the container with id c3fe2f04ee806d1967f19adf797e99f69ddf7416ac3945d2548178395d2f89d8 Jul 23 11:30:19.599328 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:19.599296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" event={"ID":"9f842103-20f5-40be-9321-067a1ff53fe2","Type":"ContainerStarted","Data":"c3fe2f04ee806d1967f19adf797e99f69ddf7416ac3945d2548178395d2f89d8"} Jul 23 11:30:20.605983 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:20.605954 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerID="507beae8f6f7864f4a899dbeda27bc30b10ac7e90ef90d78991c2d356dc2bb0e" exitCode=0 Jul 23 11:30:20.606474 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:20.606029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" event={"ID":"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f","Type":"ContainerDied","Data":"507beae8f6f7864f4a899dbeda27bc30b10ac7e90ef90d78991c2d356dc2bb0e"} Jul 23 11:30:21.307909 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:21.307873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:21.408500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:21.408463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:21.614093 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:21.614002 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerID="ffd82330ca42f247a191179e6c119a4d541a3b98149a71a8a21bb818548b010a" exitCode=0 Jul 23 11:30:21.614473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:21.614102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" event={"ID":"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f","Type":"ContainerDied","Data":"ffd82330ca42f247a191179e6c119a4d541a3b98149a71a8a21bb818548b010a"} Jul 23 11:30:22.620216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.620178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" event={"ID":"9f842103-20f5-40be-9321-067a1ff53fe2","Type":"ContainerStarted","Data":"70022677b2c8e5c1850a658884380629a44364e75848a38f6e716853a621ec3a"} Jul 23 11:30:22.637118 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.637067 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sblzv" podStartSLOduration=1.678250345 podStartE2EDuration="4.637051841s" totalImagesPullingTime="2.958801496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:22.63622511 +0000 UTC m=+216.618222625" watchObservedRunningTime="2026-07-23 11:30:22.637051841 +0000 UTC m=+216.619049343" Jul 23 11:30:22.762023 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.761999 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:22.771931 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.771892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:22.872857 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.872745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:22.943580 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.943541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-784xj\" (UniqueName: \"kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj\") pod \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " Jul 23 11:30:22.943742 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.943637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util\") pod \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " Jul 23 11:30:22.943742 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.943671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle\") pod \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\" (UID: \"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f\") " Jul 23 11:30:22.944096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.944071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle" (OuterVolumeSpecName: "bundle") pod "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" (UID: "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:22.945956 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.945931 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj" (OuterVolumeSpecName: "kube-api-access-784xj") pod "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" (UID: "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f"). InnerVolumeSpecName "kube-api-access-784xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:30:22.950076 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:22.950051 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util" (OuterVolumeSpecName: "util") pod "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" (UID: "e2ae3c30-ee70-46f7-b9eb-a16e42c9801f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:23.045122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.045084 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:23.045122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.045115 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:23.045122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.045124 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784xj\" (UniqueName: \"kubernetes.io/projected/e2ae3c30-ee70-46f7-b9eb-a16e42c9801f-kube-api-access-784xj\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:23.626861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.626823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" event={"ID":"e2ae3c30-ee70-46f7-b9eb-a16e42c9801f","Type":"ContainerDied","Data":"48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4"} Jul 23 11:30:23.626861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.626850 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjhnmj" Jul 23 11:30:23.627370 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:23.626872 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48776f929161ce77fe9acc70f56768f267436d9b5d18bb313e70b78c9bda1be4" Jul 23 11:30:25.912208 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:25.912161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:27.325827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.325783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:27.571610 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.571578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:27.818776 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.818740 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggnh7"] Jul 23 11:30:27.819057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819042 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="util" Jul 23 11:30:27.819057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819057 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="util" Jul 23 11:30:27.819184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819076 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="pull" Jul 23 11:30:27.819184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819081 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="pull" Jul 23 11:30:27.819184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819094 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="extract" Jul 23 11:30:27.819184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819100 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="extract" Jul 23 11:30:27.819184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.819144 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ae3c30-ee70-46f7-b9eb-a16e42c9801f" containerName="extract" Jul 23 11:30:27.821002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.820979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.823263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.823244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rzjrj\"" Jul 23 11:30:27.830067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.830038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggnh7"] Jul 23 11:30:27.883142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.883112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.883304 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.883156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7stg\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-kube-api-access-x7stg\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.983955 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.983917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7stg\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-kube-api-access-x7stg\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.984125 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.984009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.991889 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.991863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-bound-sa-token\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:27.992116 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:27.992096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7stg\" (UniqueName: \"kubernetes.io/projected/dfd3d8a6-6746-4561-a580-2c3f1aff317f-kube-api-access-x7stg\") pod \"cert-manager-55b4d954d5-ggnh7\" (UID: \"dfd3d8a6-6746-4561-a580-2c3f1aff317f\") " pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:28.131719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:28.131632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ggnh7" Jul 23 11:30:28.258342 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:28.258307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ggnh7"] Jul 23 11:30:28.261005 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:28.260976 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfd3d8a6_6746_4561_a580_2c3f1aff317f.slice/crio-035eb32c01d4df417b713e989024a11af7ec72cb9c7e7f9d9e827aeb7e2f892a WatchSource:0}: Error finding container 035eb32c01d4df417b713e989024a11af7ec72cb9c7e7f9d9e827aeb7e2f892a: Status 404 returned error can't find the container with id 035eb32c01d4df417b713e989024a11af7ec72cb9c7e7f9d9e827aeb7e2f892a Jul 23 11:30:28.654273 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:28.654234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ggnh7" event={"ID":"dfd3d8a6-6746-4561-a580-2c3f1aff317f","Type":"ContainerStarted","Data":"6781d6561aa01d2d1dc866a01a961757e3ea22bcaaadd4cc79fb9397871efede"} Jul 23 11:30:28.654648 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:28.654283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ggnh7" event={"ID":"dfd3d8a6-6746-4561-a580-2c3f1aff317f","Type":"ContainerStarted","Data":"035eb32c01d4df417b713e989024a11af7ec72cb9c7e7f9d9e827aeb7e2f892a"} Jul 23 11:30:28.671054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:28.670999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ggnh7" podStartSLOduration=1.6709834639999999 podStartE2EDuration="1.670983464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:28.669632609 +0000 UTC m=+222.651630116" watchObservedRunningTime="2026-07-23 11:30:28.670983464 +0000 UTC m=+222.652980967" Jul 23 11:30:29.543847 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.543808 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr"] Jul 23 11:30:29.545917 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.545893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.548601 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.548578 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 11:30:29.548693 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.548604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:30:29.549638 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.549620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6jddw\"" Jul 23 11:30:29.556657 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.556631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr"] Jul 23 11:30:29.595205 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.595163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph6mj\" (UniqueName: \"kubernetes.io/projected/70996867-f248-4731-a661-482f6cef0105-kube-api-access-ph6mj\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.595379 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.595232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70996867-f248-4731-a661-482f6cef0105-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.696189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.696155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph6mj\" (UniqueName: \"kubernetes.io/projected/70996867-f248-4731-a661-482f6cef0105-kube-api-access-ph6mj\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.696629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.696266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70996867-f248-4731-a661-482f6cef0105-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.696821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.696801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70996867-f248-4731-a661-482f6cef0105-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.705383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.705362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph6mj\" (UniqueName: \"kubernetes.io/projected/70996867-f248-4731-a661-482f6cef0105-kube-api-access-ph6mj\") pod \"openshift-lws-operator-fd8ccff4c-s4hpr\" (UID: \"70996867-f248-4731-a661-482f6cef0105\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.855514 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.855409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" Jul 23 11:30:29.992720 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:29.992696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr"] Jul 23 11:30:29.994681 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:29.994651 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70996867_f248_4731_a661_482f6cef0105.slice/crio-80df6eed1b887470b366f0f5065fbb65806ec2ce9c6a0bf6beb3c472c16e424e WatchSource:0}: Error finding container 80df6eed1b887470b366f0f5065fbb65806ec2ce9c6a0bf6beb3c472c16e424e: Status 404 returned error can't find the container with id 80df6eed1b887470b366f0f5065fbb65806ec2ce9c6a0bf6beb3c472c16e424e Jul 23 11:30:30.665757 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:30.665718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" event={"ID":"70996867-f248-4731-a661-482f6cef0105","Type":"ContainerStarted","Data":"80df6eed1b887470b366f0f5065fbb65806ec2ce9c6a0bf6beb3c472c16e424e"} Jul 23 11:30:32.677114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:32.677077 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" event={"ID":"70996867-f248-4731-a661-482f6cef0105","Type":"ContainerStarted","Data":"eb463fda2a63e987c4295622db1079dac83518f37930e1cfbf3811ffe2ad6c44"} Jul 23 11:30:32.699718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:32.699665 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s4hpr" podStartSLOduration=1.657232928 podStartE2EDuration="3.699649712s" totalImagesPullingTime="2.042416784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:32.697905375 +0000 UTC m=+226.679902881" watchObservedRunningTime="2026-07-23 11:30:32.699649712 +0000 UTC m=+226.681647214" Jul 23 11:30:33.360940 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:33.360902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:33.492697 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:33.492660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:35.045186 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:35.045152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:37.395555 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.395517 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr"] Jul 23 11:30:37.397996 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.397979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.400341 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.400312 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:30:37.400448 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.400359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jzk2n\"" Jul 23 11:30:37.400448 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.400370 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:30:37.407843 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.407818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr"] Jul 23 11:30:37.448279 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.448245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqfg\" (UniqueName: \"kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.448455 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.448292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.448455 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.448390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.548943 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.548903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqfg\" (UniqueName: \"kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.549120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.548993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.549120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.549043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.549455 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.549432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.549527 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.549467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.557650 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.557630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqfg\" (UniqueName: \"kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.707798 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.707703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:37.834988 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:37.834956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr"] Jul 23 11:30:37.837199 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:37.837161 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecd9734_af22_48b2_b779_6ec9dca51f3a.slice/crio-f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1 WatchSource:0}: Error finding container f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1: Status 404 returned error can't find the container with id f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1 Jul 23 11:30:38.704172 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:38.704136 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerID="26cea7c83d793a198aed4af5af050d6a7b87990093524b9542221bed13b336d9" exitCode=0 Jul 23 11:30:38.704647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:38.704203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" event={"ID":"4ecd9734-af22-48b2-b779-6ec9dca51f3a","Type":"ContainerDied","Data":"26cea7c83d793a198aed4af5af050d6a7b87990093524b9542221bed13b336d9"} Jul 23 11:30:38.704647 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:38.704248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" event={"ID":"4ecd9734-af22-48b2-b779-6ec9dca51f3a","Type":"ContainerStarted","Data":"f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1"} Jul 23 11:30:39.710174 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:39.710090 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerID="a8d028d466220700185d70431256e1536e283ed72744a6ffe01fef9746080a9a" exitCode=0 Jul 23 11:30:39.710174 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:39.710136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" event={"ID":"4ecd9734-af22-48b2-b779-6ec9dca51f3a","Type":"ContainerDied","Data":"a8d028d466220700185d70431256e1536e283ed72744a6ffe01fef9746080a9a"} Jul 23 11:30:40.716672 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:40.716638 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerID="01ac79d719d820cbab081e89c661a58a1a54fe13128b1aa9008a8aa02072f088" exitCode=0 Jul 23 11:30:40.717073 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:40.716737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" event={"ID":"4ecd9734-af22-48b2-b779-6ec9dca51f3a","Type":"ContainerDied","Data":"01ac79d719d820cbab081e89c661a58a1a54fe13128b1aa9008a8aa02072f088"} Jul 23 11:30:41.861193 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.861167 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:41.879309 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.879280 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knqfg\" (UniqueName: \"kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg\") pod \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " Jul 23 11:30:41.879545 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.879346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util\") pod \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " Jul 23 11:30:41.879545 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.879384 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle\") pod \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\" (UID: \"4ecd9734-af22-48b2-b779-6ec9dca51f3a\") " Jul 23 11:30:41.880754 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.880706 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle" (OuterVolumeSpecName: "bundle") pod "4ecd9734-af22-48b2-b779-6ec9dca51f3a" (UID: "4ecd9734-af22-48b2-b779-6ec9dca51f3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:41.882268 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.882239 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg" (OuterVolumeSpecName: "kube-api-access-knqfg") pod "4ecd9734-af22-48b2-b779-6ec9dca51f3a" (UID: "4ecd9734-af22-48b2-b779-6ec9dca51f3a"). InnerVolumeSpecName "kube-api-access-knqfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:30:41.885937 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.885914 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util" (OuterVolumeSpecName: "util") pod "4ecd9734-af22-48b2-b779-6ec9dca51f3a" (UID: "4ecd9734-af22-48b2-b779-6ec9dca51f3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:30:41.980190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.980113 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:41.980190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.980138 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqfg\" (UniqueName: \"kubernetes.io/projected/4ecd9734-af22-48b2-b779-6ec9dca51f3a-kube-api-access-knqfg\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:41.980190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:41.980147 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ecd9734-af22-48b2-b779-6ec9dca51f3a-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:30:42.728034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:42.727996 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" event={"ID":"4ecd9734-af22-48b2-b779-6ec9dca51f3a","Type":"ContainerDied","Data":"f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1"} Jul 23 11:30:42.728167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:42.728050 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94695811054af74ce4f56d3471e37cd9c7bd0d24fa2a9bdf719793942ba79e1" Jul 23 11:30:42.728167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:42.728016 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg4gdr" Jul 23 11:30:49.831171 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:49.831134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:49.975969 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:49.975937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:50.261297 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.261266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:50.571055 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.570972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:50.675618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.675583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:50.750412 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750375 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4"] Jul 23 11:30:50.750809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750793 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="extract" Jul 23 11:30:50.750809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750810 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="extract" Jul 23 11:30:50.750933 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750821 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="pull" Jul 23 11:30:50.750933 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750826 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="pull" Jul 23 11:30:50.750933 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750839 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="util" Jul 23 11:30:50.750933 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750844 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="util" Jul 23 11:30:50.750933 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.750898 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecd9734-af22-48b2-b779-6ec9dca51f3a" containerName="extract" Jul 23 11:30:50.758784 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.758744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.761993 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.761968 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 11:30:50.762949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.762926 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 11:30:50.763209 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.763185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5hr4d\"" Jul 23 11:30:50.763574 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.763558 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 11:30:50.767593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.767577 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 11:30:50.783661 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.783626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4"] Jul 23 11:30:50.851332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.851237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.851332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.851276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7qv\" (UniqueName: \"kubernetes.io/projected/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-kube-api-access-cp7qv\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.851332 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.851304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.851839 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.851380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.851839 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.851423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.952611 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.952583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.952611 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.952615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7qv\" (UniqueName: \"kubernetes.io/projected/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-kube-api-access-cp7qv\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.952911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.952635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.952911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.952671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.952911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.952693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.953128 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.953109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-manifests\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.955077 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.955052 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-tmp\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.955276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.955258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.955315 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.955277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:50.969914 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:50.969890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7qv\" (UniqueName: \"kubernetes.io/projected/f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7-kube-api-access-cp7qv\") pod \"opendatahub-operator-controller-manager-674bfcc66c-slnv4\" (UID: \"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7\") " pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:51.022322 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.022292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.071484 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.071440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:51.124788 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.124694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.210885 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.210852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4"] Jul 23 11:30:51.213848 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.213809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.215281 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:30:51.215259 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64dcc09_3f9a_4f11_b85e_3554a9a5cdd7.slice/crio-599db186a02e8c2baf497143fa29a2451929b53476fede482fffe24fc8f496ac WatchSource:0}: Error finding container 599db186a02e8c2baf497143fa29a2451929b53476fede482fffe24fc8f496ac: Status 404 returned error can't find the container with id 599db186a02e8c2baf497143fa29a2451929b53476fede482fffe24fc8f496ac Jul 23 11:30:51.290533 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.290492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.394996 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.394909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.493078 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.493039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:51.768699 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:51.768661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" event={"ID":"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7","Type":"ContainerStarted","Data":"599db186a02e8c2baf497143fa29a2451929b53476fede482fffe24fc8f496ac"} Jul 23 11:30:54.785284 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:54.785253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" event={"ID":"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7","Type":"ContainerStarted","Data":"a156433aa68c5f1eea40231ceef4de3e85b674dd8e72f662e0c2866aeefe3de1"} Jul 23 11:30:55.790779 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:55.790740 2564 generic.go:356] "Generic (PLEG): container finished" podID="f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7" containerID="a156433aa68c5f1eea40231ceef4de3e85b674dd8e72f662e0c2866aeefe3de1" exitCode=0 Jul 23 11:30:55.791161 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:55.790826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" event={"ID":"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7","Type":"ContainerDied","Data":"a156433aa68c5f1eea40231ceef4de3e85b674dd8e72f662e0c2866aeefe3de1"} Jul 23 11:30:56.797366 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:56.797330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" event={"ID":"f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7","Type":"ContainerStarted","Data":"9d6213f7172566e67badc956d0ef66a7f67592f7bb53be2ffe894d95e0040113"} Jul 23 11:30:56.797807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:56.797447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:30:56.822343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:56.822290 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" podStartSLOduration=1.938830554 podStartE2EDuration="6.822271114s" totalImagesPullingTime="3.703871661s" totalInitContainerRuntime="1.179568899s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:30:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:30:56.819916598 +0000 UTC m=+250.801914125" watchObservedRunningTime="2026-07-23 11:30:56.822271114 +0000 UTC m=+250.804268617" Jul 23 11:30:57.287373 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:57.287331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:30:57.401032 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:30:57.400998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:07.803882 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:07.803805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-674bfcc66c-slnv4" Jul 23 11:31:10.320964 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.320933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:10.474016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.473939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:10.767643 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.767615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:10.873751 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.873721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:10.982513 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.982481 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858"] Jul 23 11:31:10.984634 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.984618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:10.986954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.986932 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 11:31:10.988131 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.988104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 11:31:10.988131 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.988121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-66qgl\"" Jul 23 11:31:10.988270 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.988122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 11:31:10.988270 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.988222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 11:31:10.998212 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:10.998191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858"] Jul 23 11:31:11.137207 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.137124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.137207 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.137178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzt66\" (UniqueName: \"kubernetes.io/projected/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-kube-api-access-xzt66\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.137394 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.137212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tmp\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.167964 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.167933 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mm2zj"] Jul 23 11:31:11.170043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.170027 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.173004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.172984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-46cnj\"" Jul 23 11:31:11.174935 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.174915 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 11:31:11.189126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.189097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mm2zj"] Jul 23 11:31:11.237812 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.237779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.238010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.237865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzt66\" (UniqueName: \"kubernetes.io/projected/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-kube-api-access-xzt66\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.238010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.237892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tmp\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.240258 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.240232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tmp\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.240359 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.240242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-tls-certs\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.255480 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.255446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzt66\" (UniqueName: \"kubernetes.io/projected/260ad48d-5e1c-47b5-a8fe-a0cf6c233712-kube-api-access-xzt66\") pod \"kube-auth-proxy-766d5cd5c5-ph858\" (UID: \"260ad48d-5e1c-47b5-a8fe-a0cf6c233712\") " pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.259073 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.259051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:11.294616 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.294588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" Jul 23 11:31:11.339142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.339100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.339142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.339137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mcr\" (UniqueName: \"kubernetes.io/projected/fd486d6b-29c0-4374-a224-d4e83b2c4495-kube-api-access-h4mcr\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.359196 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.359137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:11.429049 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.428987 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858"] Jul 23 11:31:11.431179 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:31:11.431143 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ad48d_5e1c_47b5_a8fe_a0cf6c233712.slice/crio-61774e20ce06dcc211bd26e26870bdecdd9e4a47e49258a3ca8d90f2a90b194a WatchSource:0}: Error finding container 61774e20ce06dcc211bd26e26870bdecdd9e4a47e49258a3ca8d90f2a90b194a: Status 404 returned error can't find the container with id 61774e20ce06dcc211bd26e26870bdecdd9e4a47e49258a3ca8d90f2a90b194a Jul 23 11:31:11.439653 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.439632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.439755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.439661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mcr\" (UniqueName: \"kubernetes.io/projected/fd486d6b-29c0-4374-a224-d4e83b2c4495-kube-api-access-h4mcr\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.439836 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:31:11.439796 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 11:31:11.439873 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:31:11.439862 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert podName:fd486d6b-29c0-4374-a224-d4e83b2c4495 nodeName:}" failed. No retries permitted until 2026-07-23 11:31:11.939843956 +0000 UTC m=+265.921841439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert") pod "odh-model-controller-7998c67dc4-mm2zj" (UID: "fd486d6b-29c0-4374-a224-d4e83b2c4495") : secret "odh-model-controller-webhook-cert" not found Jul 23 11:31:11.453732 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.453712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mcr\" (UniqueName: \"kubernetes.io/projected/fd486d6b-29c0-4374-a224-d4e83b2c4495-kube-api-access-h4mcr\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.761716 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.761670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:11.864041 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.864009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" event={"ID":"260ad48d-5e1c-47b5-a8fe-a0cf6c233712","Type":"ContainerStarted","Data":"61774e20ce06dcc211bd26e26870bdecdd9e4a47e49258a3ca8d90f2a90b194a"} Jul 23 11:31:11.944327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.944291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:11.946864 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:11.946842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd486d6b-29c0-4374-a224-d4e83b2c4495-cert\") pod \"odh-model-controller-7998c67dc4-mm2zj\" (UID: \"fd486d6b-29c0-4374-a224-d4e83b2c4495\") " pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:12.079290 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.079208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:12.092928 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.092899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:12.203890 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.203830 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mvmhl"] Jul 23 11:31:12.206170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.206147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.208798 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.208621 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 11:31:12.208798 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.208647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fnf9h\"" Jul 23 11:31:12.220104 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.220082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mvmhl"] Jul 23 11:31:12.231988 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.231960 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:12.234630 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.234606 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.238434 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.237902 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4vjtg\"" Jul 23 11:31:12.238715 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.238256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 11:31:12.239083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.238613 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 11:31:12.239871 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.239727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:12.240311 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.240141 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 11:31:12.240311 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.240141 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:31:12.255794 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.255748 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:12.299389 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.299356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:12.348615 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.348615 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmrc\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5p79\" (UniqueName: \"kubernetes.io/projected/80fb3559-341d-4ea0-8596-4a6827ba824a-kube-api-access-b5p79\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fb3559-341d-4ea0-8596-4a6827ba824a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.349153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.348940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.428306 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.428260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mm2zj"] Jul 23 11:31:12.432147 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:31:12.432115 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd486d6b_29c0_4374_a224_d4e83b2c4495.slice/crio-9cf46e486309685a13e37d5f1b93de97c50a92f7840d2cc17075d08914052f25 WatchSource:0}: Error finding container 9cf46e486309685a13e37d5f1b93de97c50a92f7840d2cc17075d08914052f25: Status 404 returned error can't find the container with id 9cf46e486309685a13e37d5f1b93de97c50a92f7840d2cc17075d08914052f25 Jul 23 11:31:12.450092 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fb3559-341d-4ea0-8596-4a6827ba824a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.450190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmrc\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.450555 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5p79\" (UniqueName: \"kubernetes.io/projected/80fb3559-341d-4ea0-8596-4a6827ba824a-kube-api-access-b5p79\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.450965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.450792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.455276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.455215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fb3559-341d-4ea0-8596-4a6827ba824a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.459592 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.455812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.461986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.461958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.463323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.462595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.464371 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.464328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.464604 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.464577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5p79\" (UniqueName: \"kubernetes.io/projected/80fb3559-341d-4ea0-8596-4a6827ba824a-kube-api-access-b5p79\") pod \"model-serving-api-54d89bd5fc-mvmhl\" (UID: \"80fb3559-341d-4ea0-8596-4a6827ba824a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.465023 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.464998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.465985 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.465965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmrc\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc\") pod \"istiod-openshift-gateway-7574459656-dznqj\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.519677 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.519643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:12.557845 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.557815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:12.669577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.669541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mvmhl"] Jul 23 11:31:12.672248 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:31:12.672222 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fb3559_341d_4ea0_8596_4a6827ba824a.slice/crio-416a5c3689ce4526aeee79ed8fdc22ac02f2e5d0fcd3ef0c63ef93fb3aefb02f WatchSource:0}: Error finding container 416a5c3689ce4526aeee79ed8fdc22ac02f2e5d0fcd3ef0c63ef93fb3aefb02f: Status 404 returned error can't find the container with id 416a5c3689ce4526aeee79ed8fdc22ac02f2e5d0fcd3ef0c63ef93fb3aefb02f Jul 23 11:31:12.705426 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.705394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:12.708993 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:31:12.708959 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0369c1_0c66_4e9d_802f_630d41aae1cf.slice/crio-f2e8afe8404d4914070eafb930905ccd9d2630708c04eefd41de81922a918e8f WatchSource:0}: Error finding container f2e8afe8404d4914070eafb930905ccd9d2630708c04eefd41de81922a918e8f: Status 404 returned error can't find the container with id f2e8afe8404d4914070eafb930905ccd9d2630708c04eefd41de81922a918e8f Jul 23 11:31:12.871377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.871290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" event={"ID":"fd486d6b-29c0-4374-a224-d4e83b2c4495","Type":"ContainerStarted","Data":"9cf46e486309685a13e37d5f1b93de97c50a92f7840d2cc17075d08914052f25"} Jul 23 11:31:12.872846 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.872812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" event={"ID":"4a0369c1-0c66-4e9d-802f-630d41aae1cf","Type":"ContainerStarted","Data":"f2e8afe8404d4914070eafb930905ccd9d2630708c04eefd41de81922a918e8f"} Jul 23 11:31:12.874063 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:12.874040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" event={"ID":"80fb3559-341d-4ea0-8596-4a6827ba824a","Type":"ContainerStarted","Data":"416a5c3689ce4526aeee79ed8fdc22ac02f2e5d0fcd3ef0c63ef93fb3aefb02f"} Jul 23 11:31:13.279068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:13.278960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:15.895558 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:15.895132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" event={"ID":"260ad48d-5e1c-47b5-a8fe-a0cf6c233712","Type":"ContainerStarted","Data":"400e761e118b679d27d35e4354b2416a7f0be2f4718ed8a822d6be8ab2df879e"} Jul 23 11:31:15.897743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:15.897686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" event={"ID":"fd486d6b-29c0-4374-a224-d4e83b2c4495","Type":"ContainerStarted","Data":"ef76eba3470b815a3d5b9ddf745d84e5c137738299c654d2fa0bb2bf9177f6f5"} Jul 23 11:31:15.898000 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:15.897943 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:15.914561 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:15.913129 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-766d5cd5c5-ph858" podStartSLOduration=1.6417590469999999 podStartE2EDuration="5.913108005s" totalImagesPullingTime="4.271348958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:15.911342661 +0000 UTC m=+269.893340385" watchObservedRunningTime="2026-07-23 11:31:15.913108005 +0000 UTC m=+269.895105511" Jul 23 11:31:15.930859 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:15.929858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" podStartSLOduration=1.6121348439999998 podStartE2EDuration="4.92983416s" totalImagesPullingTime="3.317699316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:15.927666561 +0000 UTC m=+269.909664068" watchObservedRunningTime="2026-07-23 11:31:15.92983416 +0000 UTC m=+269.911831667" Jul 23 11:31:17.522849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.522744 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:31:17.522849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.522845 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:31:17.909201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.909166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" event={"ID":"80fb3559-341d-4ea0-8596-4a6827ba824a","Type":"ContainerStarted","Data":"f00295022934bfff793de7bbe22e2c5e5e789f79750754167e555edd764bea20"} Jul 23 11:31:17.909410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.909236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:17.910737 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.910713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" event={"ID":"4a0369c1-0c66-4e9d-802f-630d41aae1cf","Type":"ContainerStarted","Data":"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d"} Jul 23 11:31:17.910842 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.910826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:17.928114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.928059 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" podStartSLOduration=1.137141692 podStartE2EDuration="5.928045548s" totalImagesPullingTime="4.790903856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:17.926695059 +0000 UTC m=+271.908692563" watchObservedRunningTime="2026-07-23 11:31:17.928045548 +0000 UTC m=+271.910043049" Jul 23 11:31:17.948483 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:17.948425 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" podStartSLOduration=1.137506801 podStartE2EDuration="5.948377612s" totalImagesPullingTime="4.810870811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:17.947708198 +0000 UTC m=+271.929705702" watchObservedRunningTime="2026-07-23 11:31:17.948377612 +0000 UTC m=+271.930375116" Jul 23 11:31:18.367963 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:18.367925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:18.464063 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:18.464030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:18.921656 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:18.921595 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dznqj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 11:31:18.922126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:18.921717 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 11:31:19.060161 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:19.060124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:19.686491 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:19.686457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:19.821525 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:19.821494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:21.407303 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.407266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:21.451319 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.451286 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc"] Jul 23 11:31:21.454633 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.454615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.457334 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.457314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 11:31:21.457432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.457393 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rmdz8\"" Jul 23 11:31:21.469983 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.469957 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg"] Jul 23 11:31:21.473117 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.473100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.475569 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.475544 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 11:31:21.475683 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.475551 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mrhl5\"" Jul 23 11:31:21.477508 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.477489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc"] Jul 23 11:31:21.484646 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.484624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg"] Jul 23 11:31:21.512597 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.512565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:21.537216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.537186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39e6c624-4b32-4437-852b-561aaaf22c21-cert\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.537216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.537216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsts\" (UniqueName: \"kubernetes.io/projected/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-kube-api-access-8dsts\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.537407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.537238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.537407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.537257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2r8\" (UniqueName: \"kubernetes.io/projected/39e6c624-4b32-4437-852b-561aaaf22c21-kube-api-access-xb2r8\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.637951 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.637912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39e6c624-4b32-4437-852b-561aaaf22c21-cert\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.637951 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.637953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsts\" (UniqueName: \"kubernetes.io/projected/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-kube-api-access-8dsts\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.638210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.637989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.638210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.638020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2r8\" (UniqueName: \"kubernetes.io/projected/39e6c624-4b32-4437-852b-561aaaf22c21-kube-api-access-xb2r8\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.638210 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:31:21.638166 2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 11:31:21.638353 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:31:21.638241 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert podName:6f5ebd27-aa6d-43e9-8c2f-914ce64afa55 nodeName:}" failed. No retries permitted until 2026-07-23 11:31:22.138223765 +0000 UTC m=+276.120221263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert") pod "llmisvc-controller-manager-777c6bd7d5-8pnpg" (UID: "6f5ebd27-aa6d-43e9-8c2f-914ce64afa55") : secret "llmisvc-webhook-server-cert" not found Jul 23 11:31:21.640494 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.640472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39e6c624-4b32-4437-852b-561aaaf22c21-cert\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.658201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.658124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2r8\" (UniqueName: \"kubernetes.io/projected/39e6c624-4b32-4437-852b-561aaaf22c21-kube-api-access-xb2r8\") pod \"kserve-controller-manager-d6f4b65b8-mlxzc\" (UID: \"39e6c624-4b32-4437-852b-561aaaf22c21\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.665250 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.665223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsts\" (UniqueName: \"kubernetes.io/projected/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-kube-api-access-8dsts\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:21.705725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.705689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:21.764670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.764629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:21.878120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.878086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:21.909070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.909005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc"] Jul 23 11:31:21.919342 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.919323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:21.934337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:21.934310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" event={"ID":"39e6c624-4b32-4437-852b-561aaaf22c21","Type":"ContainerStarted","Data":"63cfeda01086fc250b31ff2b2cb5d6e19cd5a1f2a33ef26f59c4c634974bd2e1"} Jul 23 11:31:22.142527 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.142495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:22.145177 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.145152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f5ebd27-aa6d-43e9-8c2f-914ce64afa55-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8pnpg\" (UID: \"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:22.385251 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.385203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:22.529096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.529071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg"] Jul 23 11:31:22.530753 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:31:22.530713 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f5ebd27_aa6d_43e9_8c2f_914ce64afa55.slice/crio-66efc770ede3adee8128bf839d852ea4b9b0d92eb7cb19624af9e2f30abaf85c WatchSource:0}: Error finding container 66efc770ede3adee8128bf839d852ea4b9b0d92eb7cb19624af9e2f30abaf85c: Status 404 returned error can't find the container with id 66efc770ede3adee8128bf839d852ea4b9b0d92eb7cb19624af9e2f30abaf85c Jul 23 11:31:22.663227 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.663129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:22.773739 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.773709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:22.870734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.870701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:31:22.941963 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:22.941882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" event={"ID":"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55","Type":"ContainerStarted","Data":"66efc770ede3adee8128bf839d852ea4b9b0d92eb7cb19624af9e2f30abaf85c"} Jul 23 11:31:23.847269 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:23.847235 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:23.847742 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:23.847481 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerName="discovery" containerID="cri-o://99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d" gracePeriod=30 Jul 23 11:31:24.762779 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.762736 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:24.871411 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccmrc\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.871897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.871871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap\") pod \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\" (UID: \"4a0369c1-0c66-4e9d-802f-630d41aae1cf\") " Jul 23 11:31:24.872783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.872732 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:31:24.874901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.874848 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:31:24.874991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.874938 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:31:24.875069 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.875042 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs" (OuterVolumeSpecName: "local-certs") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:31:24.875423 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.875396 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:31:24.875516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.875447 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc" (OuterVolumeSpecName: "kube-api-access-ccmrc") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "kube-api-access-ccmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:31:24.875569 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.875535 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts" (OuterVolumeSpecName: "cacerts") pod "4a0369c1-0c66-4e9d-802f-630d41aae1cf" (UID: "4a0369c1-0c66-4e9d-802f-630d41aae1cf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:31:24.955247 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.955201 2564 generic.go:356] "Generic (PLEG): container finished" podID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerID="99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d" exitCode=0 Jul 23 11:31:24.955410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.955264 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" Jul 23 11:31:24.955410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.955294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" event={"ID":"4a0369c1-0c66-4e9d-802f-630d41aae1cf","Type":"ContainerDied","Data":"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d"} Jul 23 11:31:24.955410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.955383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dznqj" event={"ID":"4a0369c1-0c66-4e9d-802f-630d41aae1cf","Type":"ContainerDied","Data":"f2e8afe8404d4914070eafb930905ccd9d2630708c04eefd41de81922a918e8f"} Jul 23 11:31:24.955410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.955406 2564 scope.go:117] "RemoveContainer" containerID="99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d" Jul 23 11:31:24.972714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972687 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmrc\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-kube-api-access-ccmrc\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972717 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-ca-configmap\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972732 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0369c1-0c66-4e9d-802f-630d41aae1cf-local-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972745 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-cacerts\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972756 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972785 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-kubeconfig\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.972875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.972798 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4a0369c1-0c66-4e9d-802f-630d41aae1cf-istio-csr-dns-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:31:24.980031 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.980007 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:24.981936 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:24.981911 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dznqj"] Jul 23 11:31:25.352340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.352314 2564 scope.go:117] "RemoveContainer" containerID="99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d" Jul 23 11:31:25.352692 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:31:25.352660 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d\": container with ID starting with 99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d not found: ID does not exist" containerID="99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d" Jul 23 11:31:25.352829 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.352702 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d"} err="failed to get container status \"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d\": rpc error: code = NotFound desc = could not find container \"99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d\": container with ID starting with 99d67789c3cc875c329bf9fb19061f8143058a3b1594c66cc5e20b15ef4fdf7d not found: ID does not exist" Jul 23 11:31:25.961734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.961700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" event={"ID":"39e6c624-4b32-4437-852b-561aaaf22c21","Type":"ContainerStarted","Data":"58f0d2d98dbb4a4c07abcdcde136f3b5f3b64511304936a3443e6c36c7b4e4bb"} Jul 23 11:31:25.962232 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.961885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:31:25.963221 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.963198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" event={"ID":"6f5ebd27-aa6d-43e9-8c2f-914ce64afa55","Type":"ContainerStarted","Data":"234bc60f97cac3bbc862f9851cb7b44bac2e1e00f78f15fa247b5a63eab56819"} Jul 23 11:31:25.963339 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.963245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:25.978461 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.978417 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" podStartSLOduration=2.154034908 podStartE2EDuration="4.978403724s" totalImagesPullingTime="2.824368816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:25.976908289 +0000 UTC m=+279.958905794" watchObservedRunningTime="2026-07-23 11:31:25.978403724 +0000 UTC m=+279.960401231" Jul 23 11:31:25.993991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:25.993941 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" podStartSLOduration=2.120665681 podStartE2EDuration="4.993925841s" totalImagesPullingTime="2.87326016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:31:25.992346758 +0000 UTC m=+279.974344261" watchObservedRunningTime="2026-07-23 11:31:25.993925841 +0000 UTC m=+279.975923345" Jul 23 11:31:26.547255 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:26.547222 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" path="/var/lib/kubelet/pods/4a0369c1-0c66-4e9d-802f-630d41aae1cf/volumes" Jul 23 11:31:26.905345 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:26.905259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mm2zj" Jul 23 11:31:28.923076 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:28.923052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mvmhl" Jul 23 11:31:46.408563 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:46.408539 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:31:46.408563 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:46.408550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:31:46.413595 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:46.413573 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 11:31:56.972658 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:56.972626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8pnpg" Jul 23 11:31:56.979183 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:31:56.975682 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mlxzc" Jul 23 11:32:08.716283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:08.716248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:11.115312 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.115279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:11.879202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.879169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:11.977451 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.977418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:11.995999 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.995968 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:11.996403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.996386 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerName="discovery" Jul 23 11:32:11.996403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.996404 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerName="discovery" Jul 23 11:32:11.996541 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.996470 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0369c1-0c66-4e9d-802f-630d41aae1cf" containerName="discovery" Jul 23 11:32:11.999599 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:11.999580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:12.002207 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.002183 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 11:32:12.002520 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.002215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-lv28n\"" Jul 23 11:32:12.002520 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.002242 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 11:32:12.010901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.010880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:12.077058 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.077028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:12.081685 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.081651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcms\" (UniqueName: \"kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms\") pod \"kuadrant-operator-catalog-stfrh\" (UID: \"26fd4821-f370-4ec1-8ca3-0553edc58931\") " pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:12.182826 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.182737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcms\" (UniqueName: \"kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms\") pod \"kuadrant-operator-catalog-stfrh\" (UID: \"26fd4821-f370-4ec1-8ca3-0553edc58931\") " pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:12.190501 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.190479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcms\" (UniqueName: \"kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms\") pod \"kuadrant-operator-catalog-stfrh\" (UID: \"26fd4821-f370-4ec1-8ca3-0553edc58931\") " pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:12.310383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.310347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:12.352960 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.352934 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:12.444612 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.444558 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:12.446736 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:12.446708 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd4821_f370_4ec1_8ca3_0553edc58931.slice/crio-0483fbfbe83759f0518a3a83d4ea86b99bc01627744ba23f8d96cd619a2e1cfb WatchSource:0}: Error finding container 0483fbfbe83759f0518a3a83d4ea86b99bc01627744ba23f8d96cd619a2e1cfb: Status 404 returned error can't find the container with id 0483fbfbe83759f0518a3a83d4ea86b99bc01627744ba23f8d96cd619a2e1cfb Jul 23 11:32:12.448071 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.448052 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:32:12.563449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.563416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84sh9"] Jul 23 11:32:12.567641 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.567624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:12.574081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.574051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84sh9"] Jul 23 11:32:12.587391 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.587358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p975\" (UniqueName: \"kubernetes.io/projected/387dac0e-9014-4648-b8f0-06c3fafbaa9a-kube-api-access-2p975\") pod \"kuadrant-operator-catalog-84sh9\" (UID: \"387dac0e-9014-4648-b8f0-06c3fafbaa9a\") " pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:12.688532 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.688506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p975\" (UniqueName: \"kubernetes.io/projected/387dac0e-9014-4648-b8f0-06c3fafbaa9a-kube-api-access-2p975\") pod \"kuadrant-operator-catalog-84sh9\" (UID: \"387dac0e-9014-4648-b8f0-06c3fafbaa9a\") " pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:12.697331 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.697266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p975\" (UniqueName: \"kubernetes.io/projected/387dac0e-9014-4648-b8f0-06c3fafbaa9a-kube-api-access-2p975\") pod \"kuadrant-operator-catalog-84sh9\" (UID: \"387dac0e-9014-4648-b8f0-06c3fafbaa9a\") " pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:12.790547 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.790511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:12.878302 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.878264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:12.997516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:12.997425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:13.047921 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:13.047836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84sh9"] Jul 23 11:32:13.060696 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:13.060666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387dac0e_9014_4648_b8f0_06c3fafbaa9a.slice/crio-efffeb504b83b81c7f04be41597c0ffce1b8238992356f0ef27ff5794ec24b1c WatchSource:0}: Error finding container efffeb504b83b81c7f04be41597c0ffce1b8238992356f0ef27ff5794ec24b1c: Status 404 returned error can't find the container with id efffeb504b83b81c7f04be41597c0ffce1b8238992356f0ef27ff5794ec24b1c Jul 23 11:32:13.186978 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:13.186940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" event={"ID":"26fd4821-f370-4ec1-8ca3-0553edc58931","Type":"ContainerStarted","Data":"0483fbfbe83759f0518a3a83d4ea86b99bc01627744ba23f8d96cd619a2e1cfb"} Jul 23 11:32:13.188100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:13.188074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" event={"ID":"387dac0e-9014-4648-b8f0-06c3fafbaa9a","Type":"ContainerStarted","Data":"efffeb504b83b81c7f04be41597c0ffce1b8238992356f0ef27ff5794ec24b1c"} Jul 23 11:32:15.199662 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.199614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" event={"ID":"26fd4821-f370-4ec1-8ca3-0553edc58931","Type":"ContainerStarted","Data":"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4"} Jul 23 11:32:15.200217 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.199667 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" podUID="26fd4821-f370-4ec1-8ca3-0553edc58931" containerName="registry-server" containerID="cri-o://00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4" gracePeriod=2 Jul 23 11:32:15.201296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.201268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" event={"ID":"387dac0e-9014-4648-b8f0-06c3fafbaa9a","Type":"ContainerStarted","Data":"8139d9b60319dfa5dedcd3c66b5debe8d98b22c3cd8b9528019d9684ebc8d529"} Jul 23 11:32:15.216804 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.216741 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" podStartSLOduration=2.250940764 podStartE2EDuration="4.216726337s" totalImagesPullingTime="1.965785573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:15.214409335 +0000 UTC m=+329.196406849" watchObservedRunningTime="2026-07-23 11:32:15.216726337 +0000 UTC m=+329.198723840" Jul 23 11:32:15.229750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.229688 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" podStartSLOduration=1.87720543 podStartE2EDuration="3.229671352s" totalImagesPullingTime="1.352465922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:15.228102233 +0000 UTC m=+329.210099736" watchObservedRunningTime="2026-07-23 11:32:15.229671352 +0000 UTC m=+329.211668856" Jul 23 11:32:15.439216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.439189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:15.515625 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.515590 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlcms\" (UniqueName: \"kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms\") pod \"26fd4821-f370-4ec1-8ca3-0553edc58931\" (UID: \"26fd4821-f370-4ec1-8ca3-0553edc58931\") " Jul 23 11:32:15.517907 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.517881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms" (OuterVolumeSpecName: "kube-api-access-zlcms") pod "26fd4821-f370-4ec1-8ca3-0553edc58931" (UID: "26fd4821-f370-4ec1-8ca3-0553edc58931"). InnerVolumeSpecName "kube-api-access-zlcms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:32:15.616902 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:15.616864 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlcms\" (UniqueName: \"kubernetes.io/projected/26fd4821-f370-4ec1-8ca3-0553edc58931-kube-api-access-zlcms\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:16.207025 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.206989 2564 generic.go:356] "Generic (PLEG): container finished" podID="26fd4821-f370-4ec1-8ca3-0553edc58931" containerID="00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4" exitCode=0 Jul 23 11:32:16.207454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.207038 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" Jul 23 11:32:16.207454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.207074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" event={"ID":"26fd4821-f370-4ec1-8ca3-0553edc58931","Type":"ContainerDied","Data":"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4"} Jul 23 11:32:16.207454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.207123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stfrh" event={"ID":"26fd4821-f370-4ec1-8ca3-0553edc58931","Type":"ContainerDied","Data":"0483fbfbe83759f0518a3a83d4ea86b99bc01627744ba23f8d96cd619a2e1cfb"} Jul 23 11:32:16.207454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.207144 2564 scope.go:117] "RemoveContainer" containerID="00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4" Jul 23 11:32:16.217065 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.217048 2564 scope.go:117] "RemoveContainer" containerID="00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4" Jul 23 11:32:16.217350 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:32:16.217333 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4\": container with ID starting with 00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4 not found: ID does not exist" containerID="00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4" Jul 23 11:32:16.217397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.217360 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4"} err="failed to get container status \"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4\": rpc error: code = NotFound desc = could not find container \"00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4\": container with ID starting with 00d003ff98f8e6084ddaee45aaecfaf2296c372c0e513ca193e3555d339b48a4 not found: ID does not exist" Jul 23 11:32:16.229341 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.229307 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:16.233278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.233253 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stfrh"] Jul 23 11:32:16.542129 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:16.542097 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fd4821-f370-4ec1-8ca3-0553edc58931" path="/var/lib/kubelet/pods/26fd4821-f370-4ec1-8ca3-0553edc58931/volumes" Jul 23 11:32:22.879295 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:22.879256 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:22.879791 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:22.879311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:22.902300 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:22.902276 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:23.263364 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:23.263339 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-84sh9" Jul 23 11:32:27.602696 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.602660 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg"] Jul 23 11:32:27.603110 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.603033 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd4821-f370-4ec1-8ca3-0553edc58931" containerName="registry-server" Jul 23 11:32:27.603110 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.603045 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd4821-f370-4ec1-8ca3-0553edc58931" containerName="registry-server" Jul 23 11:32:27.603110 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.603101 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd4821-f370-4ec1-8ca3-0553edc58931" containerName="registry-server" Jul 23 11:32:27.607487 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.607468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.610078 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.610058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-45x6k\"" Jul 23 11:32:27.614061 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.614040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg"] Jul 23 11:32:27.721490 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.721458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.721654 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.721502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.721654 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.721567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd5z\" (UniqueName: \"kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.822984 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.822945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd5z\" (UniqueName: \"kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.823164 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.823014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.823164 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.823058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.823393 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.823372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.823465 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.823440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.831390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.831357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd5z\" (UniqueName: \"kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:27.917792 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:27.917695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:28.049018 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.048991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg"] Jul 23 11:32:28.051063 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:28.051030 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7e760e_6bf3_40d7_ab84_3bf455475ffc.slice/crio-9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e WatchSource:0}: Error finding container 9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e: Status 404 returned error can't find the container with id 9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e Jul 23 11:32:28.197548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.197484 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch"] Jul 23 11:32:28.200988 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.200971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.209312 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.209290 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch"] Jul 23 11:32:28.266400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.266369 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerID="0944c7f63accc4ea5b1f268631522d6c87053db53e46ede8669b2c8572f90946" exitCode=0 Jul 23 11:32:28.266585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.266420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" event={"ID":"ef7e760e-6bf3-40d7-ab84-3bf455475ffc","Type":"ContainerDied","Data":"0944c7f63accc4ea5b1f268631522d6c87053db53e46ede8669b2c8572f90946"} Jul 23 11:32:28.266585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.266457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" event={"ID":"ef7e760e-6bf3-40d7-ab84-3bf455475ffc","Type":"ContainerStarted","Data":"9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e"} Jul 23 11:32:28.328032 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.328005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.328197 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.328059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.328197 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.328140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts8f\" (UniqueName: \"kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.428828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.428799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts8f\" (UniqueName: \"kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.428990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.428851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.428990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.428885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.429194 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.429175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.429194 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.429195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.437157 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.437137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts8f\" (UniqueName: \"kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.510797 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.510750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:28.596136 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.596104 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s"] Jul 23 11:32:28.601194 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.601171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.608120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.608096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s"] Jul 23 11:32:28.648749 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.648725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch"] Jul 23 11:32:28.650470 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:28.650441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc683b81_d698_40bc_9405_1a315e533879.slice/crio-34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f WatchSource:0}: Error finding container 34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f: Status 404 returned error can't find the container with id 34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f Jul 23 11:32:28.731685 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.731656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5bc\" (UniqueName: \"kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.731834 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.731692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.731834 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.731720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.832879 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.832808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5bc\" (UniqueName: \"kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.832879 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.832848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.833070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.833041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.833191 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.833170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.833315 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.833300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.841577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.841550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5bc\" (UniqueName: \"kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:28.913800 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:28.913750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:29.003303 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.003274 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw"] Jul 23 11:32:29.007416 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.007397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.015113 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.015088 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw"] Jul 23 11:32:29.081870 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.081844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s"] Jul 23 11:32:29.083480 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:29.083420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030b492e_4ffb_4771_8c04_13d2c8568bee.slice/crio-121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe WatchSource:0}: Error finding container 121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe: Status 404 returned error can't find the container with id 121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe Jul 23 11:32:29.135833 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.135798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.135965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.135850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.135965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.135904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnrp2\" (UniqueName: \"kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.236555 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.236527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.236661 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.236578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.236661 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.236599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnrp2\" (UniqueName: \"kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.236920 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.236902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.236965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.236918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.244183 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.244165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnrp2\" (UniqueName: \"kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.272967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.272938 2564 generic.go:356] "Generic (PLEG): container finished" podID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerID="abedf607008f987c6c5161c2e2a4a70495acbb3ba463a7c6fdc20a52b9918dba" exitCode=0 Jul 23 11:32:29.273060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.273023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" event={"ID":"030b492e-4ffb-4771-8c04-13d2c8568bee","Type":"ContainerDied","Data":"abedf607008f987c6c5161c2e2a4a70495acbb3ba463a7c6fdc20a52b9918dba"} Jul 23 11:32:29.273147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.273059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" event={"ID":"030b492e-4ffb-4771-8c04-13d2c8568bee","Type":"ContainerStarted","Data":"121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe"} Jul 23 11:32:29.274636 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.274616 2564 generic.go:356] "Generic (PLEG): container finished" podID="bc683b81-d698-40bc-9405-1a315e533879" containerID="585f8a402f959e5e1cb8d3f3f241cb8e02ed5505086c55bba66a6671f7100e87" exitCode=0 Jul 23 11:32:29.274714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.274698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" event={"ID":"bc683b81-d698-40bc-9405-1a315e533879","Type":"ContainerDied","Data":"585f8a402f959e5e1cb8d3f3f241cb8e02ed5505086c55bba66a6671f7100e87"} Jul 23 11:32:29.274780 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.274733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" event={"ID":"bc683b81-d698-40bc-9405-1a315e533879","Type":"ContainerStarted","Data":"34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f"} Jul 23 11:32:29.276707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.276588 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerID="b2b1a6bc935f8a62a93f449365177cb39d321272e26950799451cf7aecf4be39" exitCode=0 Jul 23 11:32:29.276707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.276629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" event={"ID":"ef7e760e-6bf3-40d7-ab84-3bf455475ffc","Type":"ContainerDied","Data":"b2b1a6bc935f8a62a93f449365177cb39d321272e26950799451cf7aecf4be39"} Jul 23 11:32:29.338290 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.338234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:29.477543 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:29.477518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw"] Jul 23 11:32:29.478940 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:29.478912 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66bcd69_3978_4be5_b0d2_010f16e3bf78.slice/crio-70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f WatchSource:0}: Error finding container 70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f: Status 404 returned error can't find the container with id 70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f Jul 23 11:32:30.283057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.283021 2564 generic.go:356] "Generic (PLEG): container finished" podID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerID="6c17073f78590f258ae59193e7d982172c6e7d1a3a8e3234385a42d555c3662f" exitCode=0 Jul 23 11:32:30.283430 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.283081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" event={"ID":"030b492e-4ffb-4771-8c04-13d2c8568bee","Type":"ContainerDied","Data":"6c17073f78590f258ae59193e7d982172c6e7d1a3a8e3234385a42d555c3662f"} Jul 23 11:32:30.284995 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.284974 2564 generic.go:356] "Generic (PLEG): container finished" podID="bc683b81-d698-40bc-9405-1a315e533879" containerID="746a9ab8278c47bf4e8c54dcddd3470b23646eb6290b009695e0c86ae8ae8cb8" exitCode=0 Jul 23 11:32:30.285092 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.285047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" event={"ID":"bc683b81-d698-40bc-9405-1a315e533879","Type":"ContainerDied","Data":"746a9ab8278c47bf4e8c54dcddd3470b23646eb6290b009695e0c86ae8ae8cb8"} Jul 23 11:32:30.287312 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.287291 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerID="02ba83eb467c3ed0f0622ef652892fe6360940f1bdc06fb5abc937e2c3150fb3" exitCode=0 Jul 23 11:32:30.287403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.287364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" event={"ID":"ef7e760e-6bf3-40d7-ab84-3bf455475ffc","Type":"ContainerDied","Data":"02ba83eb467c3ed0f0622ef652892fe6360940f1bdc06fb5abc937e2c3150fb3"} Jul 23 11:32:30.288904 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.288885 2564 generic.go:356] "Generic (PLEG): container finished" podID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerID="3f3fde0eebe794b14f581bc11f982d01661dc77e6c36f8090fbb7669d32d0296" exitCode=0 Jul 23 11:32:30.288981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.288941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" event={"ID":"c66bcd69-3978-4be5-b0d2-010f16e3bf78","Type":"ContainerDied","Data":"3f3fde0eebe794b14f581bc11f982d01661dc77e6c36f8090fbb7669d32d0296"} Jul 23 11:32:30.288981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.288969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" event={"ID":"c66bcd69-3978-4be5-b0d2-010f16e3bf78","Type":"ContainerStarted","Data":"70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f"} Jul 23 11:32:30.631313 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.631228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:30.732258 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:30.732219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:31.299500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.299462 2564 generic.go:356] "Generic (PLEG): container finished" podID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerID="3bfbe469a778767b79a7df3ea1022633018c8731e18390246ecc8c31ce0ed92a" exitCode=0 Jul 23 11:32:31.299952 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.299536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" event={"ID":"c66bcd69-3978-4be5-b0d2-010f16e3bf78","Type":"ContainerDied","Data":"3bfbe469a778767b79a7df3ea1022633018c8731e18390246ecc8c31ce0ed92a"} Jul 23 11:32:31.302125 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.302101 2564 generic.go:356] "Generic (PLEG): container finished" podID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerID="549cd2a67013d5332b0e0ab46ece8dc6e3642d1133612b01c8229023844197bd" exitCode=0 Jul 23 11:32:31.302209 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.302184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" event={"ID":"030b492e-4ffb-4771-8c04-13d2c8568bee","Type":"ContainerDied","Data":"549cd2a67013d5332b0e0ab46ece8dc6e3642d1133612b01c8229023844197bd"} Jul 23 11:32:31.304213 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.304186 2564 generic.go:356] "Generic (PLEG): container finished" podID="bc683b81-d698-40bc-9405-1a315e533879" containerID="a906cbb0c164e692ab69871c89b38ee3d7b18d3d15af5568ef95efb3b912f6ae" exitCode=0 Jul 23 11:32:31.304300 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.304250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" event={"ID":"bc683b81-d698-40bc-9405-1a315e533879","Type":"ContainerDied","Data":"a906cbb0c164e692ab69871c89b38ee3d7b18d3d15af5568ef95efb3b912f6ae"} Jul 23 11:32:31.433855 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.433826 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:31.558727 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.558686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsd5z\" (UniqueName: \"kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z\") pod \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " Jul 23 11:32:31.558918 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.558748 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util\") pod \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " Jul 23 11:32:31.558918 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.558827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle\") pod \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\" (UID: \"ef7e760e-6bf3-40d7-ab84-3bf455475ffc\") " Jul 23 11:32:31.559319 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.559294 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle" (OuterVolumeSpecName: "bundle") pod "ef7e760e-6bf3-40d7-ab84-3bf455475ffc" (UID: "ef7e760e-6bf3-40d7-ab84-3bf455475ffc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:31.560928 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.560909 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z" (OuterVolumeSpecName: "kube-api-access-xsd5z") pod "ef7e760e-6bf3-40d7-ab84-3bf455475ffc" (UID: "ef7e760e-6bf3-40d7-ab84-3bf455475ffc"). InnerVolumeSpecName "kube-api-access-xsd5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:32:31.563482 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.563461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util" (OuterVolumeSpecName: "util") pod "ef7e760e-6bf3-40d7-ab84-3bf455475ffc" (UID: "ef7e760e-6bf3-40d7-ab84-3bf455475ffc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:31.659757 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.659684 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsd5z\" (UniqueName: \"kubernetes.io/projected/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-kube-api-access-xsd5z\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:31.659757 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.659713 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:31.659757 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:31.659724 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef7e760e-6bf3-40d7-ab84-3bf455475ffc-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.311901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.311868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" event={"ID":"ef7e760e-6bf3-40d7-ab84-3bf455475ffc","Type":"ContainerDied","Data":"9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e"} Jul 23 11:32:32.312351 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.311923 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee6c50f893a008122316eb3eb37eb2fd986ee1ceda959f561d3933aa1cb513e" Jul 23 11:32:32.312351 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.311877 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg" Jul 23 11:32:32.314054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.314027 2564 generic.go:356] "Generic (PLEG): container finished" podID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerID="9dccee993202f7e91311c9302a05c2fecd35f787ee81b444553da57201d5dca7" exitCode=0 Jul 23 11:32:32.314144 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.314099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" event={"ID":"c66bcd69-3978-4be5-b0d2-010f16e3bf78","Type":"ContainerDied","Data":"9dccee993202f7e91311c9302a05c2fecd35f787ee81b444553da57201d5dca7"} Jul 23 11:32:32.444440 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.444417 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:32.476043 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.476021 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:32.567525 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567429 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p5bc\" (UniqueName: \"kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc\") pod \"030b492e-4ffb-4771-8c04-13d2c8568bee\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " Jul 23 11:32:32.567525 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle\") pod \"030b492e-4ffb-4771-8c04-13d2c8568bee\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " Jul 23 11:32:32.567525 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle\") pod \"bc683b81-d698-40bc-9405-1a315e533879\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " Jul 23 11:32:32.567799 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567532 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lts8f\" (UniqueName: \"kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f\") pod \"bc683b81-d698-40bc-9405-1a315e533879\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " Jul 23 11:32:32.567799 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:32.567799 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567627 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util\") pod \"030b492e-4ffb-4771-8c04-13d2c8568bee\" (UID: \"030b492e-4ffb-4771-8c04-13d2c8568bee\") " Jul 23 11:32:32.567799 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.567654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util\") pod \"bc683b81-d698-40bc-9405-1a315e533879\" (UID: \"bc683b81-d698-40bc-9405-1a315e533879\") " Jul 23 11:32:32.568189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.568127 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle" (OuterVolumeSpecName: "bundle") pod "bc683b81-d698-40bc-9405-1a315e533879" (UID: "bc683b81-d698-40bc-9405-1a315e533879"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:32.568616 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.568595 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle" (OuterVolumeSpecName: "bundle") pod "030b492e-4ffb-4771-8c04-13d2c8568bee" (UID: "030b492e-4ffb-4771-8c04-13d2c8568bee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:32.569794 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.569753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f" (OuterVolumeSpecName: "kube-api-access-lts8f") pod "bc683b81-d698-40bc-9405-1a315e533879" (UID: "bc683b81-d698-40bc-9405-1a315e533879"). InnerVolumeSpecName "kube-api-access-lts8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:32:32.569871 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.569829 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc" (OuterVolumeSpecName: "kube-api-access-8p5bc") pod "030b492e-4ffb-4771-8c04-13d2c8568bee" (UID: "030b492e-4ffb-4771-8c04-13d2c8568bee"). InnerVolumeSpecName "kube-api-access-8p5bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:32:32.573624 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.573601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util" (OuterVolumeSpecName: "util") pod "030b492e-4ffb-4771-8c04-13d2c8568bee" (UID: "030b492e-4ffb-4771-8c04-13d2c8568bee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:32.573674 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.573650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util" (OuterVolumeSpecName: "util") pod "bc683b81-d698-40bc-9405-1a315e533879" (UID: "bc683b81-d698-40bc-9405-1a315e533879"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:32.624187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.624149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:32.668991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.668962 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.668991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.668988 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.669163 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.669002 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5bc\" (UniqueName: \"kubernetes.io/projected/030b492e-4ffb-4771-8c04-13d2c8568bee-kube-api-access-8p5bc\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.669163 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.669011 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030b492e-4ffb-4771-8c04-13d2c8568bee-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.669163 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.669021 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc683b81-d698-40bc-9405-1a315e533879-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:32.669163 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:32.669029 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lts8f\" (UniqueName: \"kubernetes.io/projected/bc683b81-d698-40bc-9405-1a315e533879-kube-api-access-lts8f\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:33.321214 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.321182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" event={"ID":"030b492e-4ffb-4771-8c04-13d2c8568bee","Type":"ContainerDied","Data":"121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe"} Jul 23 11:32:33.321650 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.321228 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="121a58be09e9dd322b68d3a3abd62ac3144abf17e66e3c322786871e356564fe" Jul 23 11:32:33.321650 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.321197 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s" Jul 23 11:32:33.323147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.323126 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" Jul 23 11:32:33.323266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.323162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch" event={"ID":"bc683b81-d698-40bc-9405-1a315e533879","Type":"ContainerDied","Data":"34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f"} Jul 23 11:32:33.323266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.323203 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a4f6f94522be32b1f8c3e3b5b63d27e2ace023967b04d9bfa55f73cd8b360f" Jul 23 11:32:33.457424 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.457401 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:33.577486 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.577398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle\") pod \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " Jul 23 11:32:33.577486 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.577447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnrp2\" (UniqueName: \"kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2\") pod \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " Jul 23 11:32:33.577664 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.577497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util\") pod \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\" (UID: \"c66bcd69-3978-4be5-b0d2-010f16e3bf78\") " Jul 23 11:32:33.578187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.578157 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle" (OuterVolumeSpecName: "bundle") pod "c66bcd69-3978-4be5-b0d2-010f16e3bf78" (UID: "c66bcd69-3978-4be5-b0d2-010f16e3bf78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:33.579691 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.579666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2" (OuterVolumeSpecName: "kube-api-access-cnrp2") pod "c66bcd69-3978-4be5-b0d2-010f16e3bf78" (UID: "c66bcd69-3978-4be5-b0d2-010f16e3bf78"). InnerVolumeSpecName "kube-api-access-cnrp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:32:33.583162 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.583130 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util" (OuterVolumeSpecName: "util") pod "c66bcd69-3978-4be5-b0d2-010f16e3bf78" (UID: "c66bcd69-3978-4be5-b0d2-010f16e3bf78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:32:33.678989 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.678954 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:33.678989 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.678991 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnrp2\" (UniqueName: \"kubernetes.io/projected/c66bcd69-3978-4be5-b0d2-010f16e3bf78-kube-api-access-cnrp2\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:33.679184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:33.679008 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66bcd69-3978-4be5-b0d2-010f16e3bf78-util\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:32:34.330302 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:34.330268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" event={"ID":"c66bcd69-3978-4be5-b0d2-010f16e3bf78","Type":"ContainerDied","Data":"70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f"} Jul 23 11:32:34.330673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:34.330314 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e31626a1767387151497a83fc9dd631410f8a2ef7b4097d7da7dab65f0559f" Jul 23 11:32:34.330673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:34.330279 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw" Jul 23 11:32:41.053240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:41.053207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:43.621017 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.620979 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k"] Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621315 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621335 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="pull" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621340 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="pull" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621349 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="pull" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621355 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="pull" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621362 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621368 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621381 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621391 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621397 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="extract" Jul 23 11:32:43.621397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621404 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="pull" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621409 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="pull" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621415 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621419 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621427 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621431 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621436 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621440 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621445 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="pull" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621450 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="pull" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621457 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621461 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="util" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621509 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc683b81-d698-40bc-9405-1a315e533879" containerName="extract" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621517 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66bcd69-3978-4be5-b0d2-010f16e3bf78" containerName="extract" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621522 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="030b492e-4ffb-4771-8c04-13d2c8568bee" containerName="extract" Jul 23 11:32:43.621888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.621531 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7e760e-6bf3-40d7-ab84-3bf455475ffc" containerName="extract" Jul 23 11:32:43.624600 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.624584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:43.627286 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.627265 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-z2rkw\"" Jul 23 11:32:43.627438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.627415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 23 11:32:43.634842 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.634815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k"] Jul 23 11:32:43.763912 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.763873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rl8\" (UniqueName: \"kubernetes.io/projected/b8a63fe9-3653-4352-8268-fda4bf5f2d68-kube-api-access-b6rl8\") pod \"dns-operator-controller-manager-864dd78c4c-p5g8k\" (UID: \"b8a63fe9-3653-4352-8268-fda4bf5f2d68\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:43.865141 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.865102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rl8\" (UniqueName: \"kubernetes.io/projected/b8a63fe9-3653-4352-8268-fda4bf5f2d68-kube-api-access-b6rl8\") pod \"dns-operator-controller-manager-864dd78c4c-p5g8k\" (UID: \"b8a63fe9-3653-4352-8268-fda4bf5f2d68\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:43.877040 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.876969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rl8\" (UniqueName: \"kubernetes.io/projected/b8a63fe9-3653-4352-8268-fda4bf5f2d68-kube-api-access-b6rl8\") pod \"dns-operator-controller-manager-864dd78c4c-p5g8k\" (UID: \"b8a63fe9-3653-4352-8268-fda4bf5f2d68\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:43.935561 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:43.935517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:44.072425 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:44.072400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k"] Jul 23 11:32:44.074410 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:44.074382 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a63fe9_3653_4352_8268_fda4bf5f2d68.slice/crio-699ebd098e199a62388566447b02407454614ab4fe4f2784870daea49cd486ae WatchSource:0}: Error finding container 699ebd098e199a62388566447b02407454614ab4fe4f2784870daea49cd486ae: Status 404 returned error can't find the container with id 699ebd098e199a62388566447b02407454614ab4fe4f2784870daea49cd486ae Jul 23 11:32:44.381088 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:44.381044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" event={"ID":"b8a63fe9-3653-4352-8268-fda4bf5f2d68","Type":"ContainerStarted","Data":"699ebd098e199a62388566447b02407454614ab4fe4f2784870daea49cd486ae"} Jul 23 11:32:44.580743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:44.580707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:46.394977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:46.394940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" event={"ID":"b8a63fe9-3653-4352-8268-fda4bf5f2d68","Type":"ContainerStarted","Data":"d5826c998ab0dbb291da19a921c2119a61b712c4a5a2ec5b8bd25d80c8f567bb"} Jul 23 11:32:46.394977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:46.394999 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:46.414529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:46.414481 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" podStartSLOduration=1.529340109 podStartE2EDuration="3.414467106s" totalImagesPullingTime="1.885126997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:46.413608125 +0000 UTC m=+360.395605642" watchObservedRunningTime="2026-07-23 11:32:46.414467106 +0000 UTC m=+360.396464608" Jul 23 11:32:47.614713 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:47.614677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:48.931537 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:48.931488 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2"] Jul 23 11:32:48.937547 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:48.937527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:48.939967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:48.939945 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jhmwr\"" Jul 23 11:32:48.949462 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:48.949441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2"] Jul 23 11:32:49.097570 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.097538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:49.111650 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.111617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.111840 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.111670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bfk\" (UniqueName: \"kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.213042 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.212949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.213042 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.213005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bfk\" (UniqueName: \"kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.213386 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.213358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.227629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.227606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bfk\" (UniqueName: \"kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.247463 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.247436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:49.385476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.385446 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2"] Jul 23 11:32:49.388141 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:49.388115 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba7176a_ca26_441a_9152_d6433b77d692.slice/crio-08dce6c1ecffb8c6b6a37cff1e35a15123167254e900467490c6f6f37363ba50 WatchSource:0}: Error finding container 08dce6c1ecffb8c6b6a37cff1e35a15123167254e900467490c6f6f37363ba50: Status 404 returned error can't find the container with id 08dce6c1ecffb8c6b6a37cff1e35a15123167254e900467490c6f6f37363ba50 Jul 23 11:32:49.410167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.410138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" event={"ID":"7ba7176a-ca26-441a-9152-d6433b77d692","Type":"ContainerStarted","Data":"08dce6c1ecffb8c6b6a37cff1e35a15123167254e900467490c6f6f37363ba50"} Jul 23 11:32:49.983972 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:49.983923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:51.924914 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:51.924882 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l"] Jul 23 11:32:51.928564 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:51.928546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:51.931266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:51.931246 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vg2gp\"" Jul 23 11:32:51.939652 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:51.939627 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l"] Jul 23 11:32:52.038181 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.038151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkl4\" (UniqueName: \"kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4\") pod \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" (UID: \"2d4667d3-0e1a-4271-9a83-8f714bcc6b26\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:52.139468 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.139356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkl4\" (UniqueName: \"kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4\") pod \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" (UID: \"2d4667d3-0e1a-4271-9a83-8f714bcc6b26\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:52.155697 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.155663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkl4\" (UniqueName: \"kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4\") pod \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" (UID: \"2d4667d3-0e1a-4271-9a83-8f714bcc6b26\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:52.175635 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.175537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:52.239631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.239596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:52.382435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.382408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l"] Jul 23 11:32:52.384496 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:52.384471 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4667d3_0e1a_4271_9a83_8f714bcc6b26.slice/crio-768b6f4c9711602282d553d44ab3c7d23936aebdd2af042d37d9f9dfa717c0ed WatchSource:0}: Error finding container 768b6f4c9711602282d553d44ab3c7d23936aebdd2af042d37d9f9dfa717c0ed: Status 404 returned error can't find the container with id 768b6f4c9711602282d553d44ab3c7d23936aebdd2af042d37d9f9dfa717c0ed Jul 23 11:32:52.428330 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.428246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" event={"ID":"2d4667d3-0e1a-4271-9a83-8f714bcc6b26","Type":"ContainerStarted","Data":"768b6f4c9711602282d553d44ab3c7d23936aebdd2af042d37d9f9dfa717c0ed"} Jul 23 11:32:52.592423 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.592394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:52.887324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:52.887291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:54.442840 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:54.442738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" event={"ID":"2d4667d3-0e1a-4271-9a83-8f714bcc6b26","Type":"ContainerStarted","Data":"91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20"} Jul 23 11:32:54.443229 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:54.442916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:32:54.463134 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:54.463081 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" podStartSLOduration=1.7581116030000001 podStartE2EDuration="3.463066899s" totalImagesPullingTime="1.704955296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:54.461653032 +0000 UTC m=+368.443650534" watchObservedRunningTime="2026-07-23 11:32:54.463066899 +0000 UTC m=+368.445064402" Jul 23 11:32:55.141822 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.141786 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq"] Jul 23 11:32:55.145611 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.145576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:55.148300 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.148279 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7dql6\"" Jul 23 11:32:55.161655 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.161625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq"] Jul 23 11:32:55.167142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.167085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56lr\" (UniqueName: \"kubernetes.io/projected/d00cd354-8afa-434c-b778-64d0475268d1-kube-api-access-p56lr\") pod \"authorino-operator-75cc8d5cb7-fkvmq\" (UID: \"d00cd354-8afa-434c-b778-64d0475268d1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:55.268200 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.268168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56lr\" (UniqueName: \"kubernetes.io/projected/d00cd354-8afa-434c-b778-64d0475268d1-kube-api-access-p56lr\") pod \"authorino-operator-75cc8d5cb7-fkvmq\" (UID: \"d00cd354-8afa-434c-b778-64d0475268d1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:55.276570 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.276546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56lr\" (UniqueName: \"kubernetes.io/projected/d00cd354-8afa-434c-b778-64d0475268d1-kube-api-access-p56lr\") pod \"authorino-operator-75cc8d5cb7-fkvmq\" (UID: \"d00cd354-8afa-434c-b778-64d0475268d1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:55.457337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.457248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:55.626680 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:55.626651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:56.218571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:56.218546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq"] Jul 23 11:32:56.220355 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:56.220333 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00cd354_8afa_434c_b778_64d0475268d1.slice/crio-eb6278dbd31ef51539e818e9f31946da56583ecacef899d58d7a04b5a1231973 WatchSource:0}: Error finding container eb6278dbd31ef51539e818e9f31946da56583ecacef899d58d7a04b5a1231973: Status 404 returned error can't find the container with id eb6278dbd31ef51539e818e9f31946da56583ecacef899d58d7a04b5a1231973 Jul 23 11:32:56.455346 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:56.455257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" event={"ID":"d00cd354-8afa-434c-b778-64d0475268d1","Type":"ContainerStarted","Data":"eb6278dbd31ef51539e818e9f31946da56583ecacef899d58d7a04b5a1231973"} Jul 23 11:32:56.456658 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:56.456630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" event={"ID":"7ba7176a-ca26-441a-9152-d6433b77d692","Type":"ContainerStarted","Data":"26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0"} Jul 23 11:32:56.456802 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:56.456731 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:32:56.482052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:56.482006 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" podStartSLOduration=1.748827789 podStartE2EDuration="8.481991208s" totalImagesPullingTime="6.733163419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:56.480032951 +0000 UTC m=+370.462030453" watchObservedRunningTime="2026-07-23 11:32:56.481991208 +0000 UTC m=+370.463988709" Jul 23 11:32:57.010181 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:57.010140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:57.183852 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:57.183791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:57.402901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:57.402818 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5g8k" Jul 23 11:32:59.173722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.173687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk"] Jul 23 11:32:59.177449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.177429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.179916 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.179892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 23 11:32:59.179916 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.179915 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-45x6k\"" Jul 23 11:32:59.180098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.179956 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 23 11:32:59.189384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.189360 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk"] Jul 23 11:32:59.193229 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.193206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:59.202986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.202959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3b996e-4708-4a70-ba2f-f5567cd666a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.203120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.203030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a3b996e-4708-4a70-ba2f-f5567cd666a8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.203120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.203103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flls6\" (UniqueName: \"kubernetes.io/projected/9a3b996e-4708-4a70-ba2f-f5567cd666a8-kube-api-access-flls6\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.271705 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.271674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:32:59.303955 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.303921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3b996e-4708-4a70-ba2f-f5567cd666a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.304123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.303979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a3b996e-4708-4a70-ba2f-f5567cd666a8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.304123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.304030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flls6\" (UniqueName: \"kubernetes.io/projected/9a3b996e-4708-4a70-ba2f-f5567cd666a8-kube-api-access-flls6\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.304648 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.304623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a3b996e-4708-4a70-ba2f-f5567cd666a8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.306571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.306547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3b996e-4708-4a70-ba2f-f5567cd666a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.317248 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.317222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flls6\" (UniqueName: \"kubernetes.io/projected/9a3b996e-4708-4a70-ba2f-f5567cd666a8-kube-api-access-flls6\") pod \"kuadrant-console-plugin-66b5774985-9sqzk\" (UID: \"9a3b996e-4708-4a70-ba2f-f5567cd666a8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.476474 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.476445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" event={"ID":"d00cd354-8afa-434c-b778-64d0475268d1","Type":"ContainerStarted","Data":"5ea9082a45926000ccec7d327132bd5dc20e5df676b740e207e8e98acd7de1e7"} Jul 23 11:32:59.476673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.476590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:32:59.486710 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.486685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" Jul 23 11:32:59.496267 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.496223 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" podStartSLOduration=2.267460464 podStartE2EDuration="4.496207894s" totalImagesPullingTime="2.22874743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:32:59.493843479 +0000 UTC m=+373.475840982" watchObservedRunningTime="2026-07-23 11:32:59.496207894 +0000 UTC m=+373.478205448" Jul 23 11:32:59.646116 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:32:59.646094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk"] Jul 23 11:32:59.648004 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:32:59.647980 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a3b996e_4708_4a70_ba2f_f5567cd666a8.slice/crio-80a42de3eba6530852fcbcefc6c8e6a4a12497cf8c4dfcbde40a07543314455f WatchSource:0}: Error finding container 80a42de3eba6530852fcbcefc6c8e6a4a12497cf8c4dfcbde40a07543314455f: Status 404 returned error can't find the container with id 80a42de3eba6530852fcbcefc6c8e6a4a12497cf8c4dfcbde40a07543314455f Jul 23 11:33:00.487864 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:00.487826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" event={"ID":"9a3b996e-4708-4a70-ba2f-f5567cd666a8","Type":"ContainerStarted","Data":"80a42de3eba6530852fcbcefc6c8e6a4a12497cf8c4dfcbde40a07543314455f"} Jul 23 11:33:00.591867 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:00.591827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:00.696738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:00.696702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:05.451863 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:05.451830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:07.396715 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:07.396686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:07.465726 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:07.465690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:07.498199 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:07.498133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:09.126075 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.126037 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2"] Jul 23 11:33:09.126491 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.126237 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" containerName="manager" containerID="cri-o://26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0" gracePeriod=2 Jul 23 11:33:09.138259 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.138234 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2"] Jul 23 11:33:09.153717 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.153670 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:09.154185 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.154168 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" containerName="manager" Jul 23 11:33:09.154264 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.154190 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" containerName="manager" Jul 23 11:33:09.154321 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.154271 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" containerName="manager" Jul 23 11:33:09.155266 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:09.155212 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7ba7176a-ca26-441a-9152-d6433b77d692\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\": pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:09.157240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.157216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.160156 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:09.160011 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:09.160156 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.160041 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l"] Jul 23 11:33:09.160318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.160230 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" containerName="manager" containerID="cri-o://91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20" gracePeriod=2 Jul 23 11:33:09.178161 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.178129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:09.180554 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.180532 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l"] Jul 23 11:33:09.190355 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.190334 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k"] Jul 23 11:33:09.190838 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.190818 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" containerName="manager" Jul 23 11:33:09.190932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.190841 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" containerName="manager" Jul 23 11:33:09.190932 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.190924 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" containerName="manager" Jul 23 11:33:09.193202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.193182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:09.200526 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:09.200449 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:09.202371 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.202334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.203061 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.202381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftrr\" (UniqueName: \"kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.205417 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.205384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k"] Jul 23 11:33:09.219562 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:09.219522 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:09.303878 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.303836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltw8\" (UniqueName: \"kubernetes.io/projected/a606d3b3-4b05-4ec9-ba75-eefd0736f411-kube-api-access-tltw8\") pod \"limitador-operator-controller-manager-859d7b78bd-cbx5k\" (UID: \"a606d3b3-4b05-4ec9-ba75-eefd0736f411\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:09.304047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.303954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.304047 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.304000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftrr\" (UniqueName: \"kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.304267 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.304239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.318755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.318723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftrr\" (UniqueName: \"kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4nq9l\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.405353 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.405267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltw8\" (UniqueName: \"kubernetes.io/projected/a606d3b3-4b05-4ec9-ba75-eefd0736f411-kube-api-access-tltw8\") pod \"limitador-operator-controller-manager-859d7b78bd-cbx5k\" (UID: \"a606d3b3-4b05-4ec9-ba75-eefd0736f411\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:09.418986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.418952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltw8\" (UniqueName: \"kubernetes.io/projected/a606d3b3-4b05-4ec9-ba75-eefd0736f411-kube-api-access-tltw8\") pod \"limitador-operator-controller-manager-859d7b78bd-cbx5k\" (UID: \"a606d3b3-4b05-4ec9-ba75-eefd0736f411\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:09.553962 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.553932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:09.560561 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:09.560526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:10.491036 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:10.491009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkvmq" Jul 23 11:33:10.493389 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:10.493358 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:10.522661 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:10.522612 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:12.204398 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:12.204352 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:12.316977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:12.316936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:14.957953 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:14.957909 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d8d5c76b-cq2z2"] Jul 23 11:33:15.024515 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.024485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:15.039308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.039282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d8d5c76b-cq2z2"] Jul 23 11:33:15.039456 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.039396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163178 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-oauth-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-service-ca\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd97q\" (UniqueName: \"kubernetes.io/projected/c9761335-16b5-4a27-8f71-1d18e0c7d871-kube-api-access-fd97q\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-oauth-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.163360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.163327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-trusted-ca-bundle\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264111 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-service-ca\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd97q\" (UniqueName: \"kubernetes.io/projected/c9761335-16b5-4a27-8f71-1d18e0c7d871-kube-api-access-fd97q\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-oauth-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-trusted-ca-bundle\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264518 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-oauth-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264518 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.264878 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.265016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.264896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-service-ca\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.265201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.265168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-oauth-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.265310 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.265263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9761335-16b5-4a27-8f71-1d18e0c7d871-trusted-ca-bundle\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.266997 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.266979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-serving-cert\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.267237 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.267219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9761335-16b5-4a27-8f71-1d18e0c7d871-console-oauth-config\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.273302 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.273278 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd97q\" (UniqueName: \"kubernetes.io/projected/c9761335-16b5-4a27-8f71-1d18e0c7d871-kube-api-access-fd97q\") pod \"console-74d8d5c76b-cq2z2\" (UID: \"c9761335-16b5-4a27-8f71-1d18e0c7d871\") " pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:15.352324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:15.352279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:17.979956 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:17.979916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:18.080151 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:18.080114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:21.116625 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.116601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d8d5c76b-cq2z2"] Jul 23 11:33:21.119301 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:21.119273 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9761335_16b5_4a27_8f71_1d18e0c7d871.slice/crio-8407ce762620d0046728f5528e5ffb4a4eb47ca7a14ef7b854378beec9bc0228 WatchSource:0}: Error finding container 8407ce762620d0046728f5528e5ffb4a4eb47ca7a14ef7b854378beec9bc0228: Status 404 returned error can't find the container with id 8407ce762620d0046728f5528e5ffb4a4eb47ca7a14ef7b854378beec9bc0228 Jul 23 11:33:21.127827 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.127805 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.129899 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.129867 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.132327 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.132304 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:21.134227 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.134193 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:21.136397 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.135890 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.147083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.147056 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:21.148152 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:21.148118 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0466c8b5_9683_471f_bcad_8c14c0c3d1a7.slice/crio-b447bb6f727ce856e98bc5749b99d1b3a1a8f0ceefe0f17b882f8172d1572993 WatchSource:0}: Error finding container b447bb6f727ce856e98bc5749b99d1b3a1a8f0ceefe0f17b882f8172d1572993: Status 404 returned error can't find the container with id b447bb6f727ce856e98bc5749b99d1b3a1a8f0ceefe0f17b882f8172d1572993 Jul 23 11:33:21.169570 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.169548 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k"] Jul 23 11:33:21.170731 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:21.170711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda606d3b3_4b05_4ec9_ba75_eefd0736f411.slice/crio-e36876a97fb3a6518d03c7ef576d921f199234b09e245ea6e63982d104e4aa9d WatchSource:0}: Error finding container e36876a97fb3a6518d03c7ef576d921f199234b09e245ea6e63982d104e4aa9d: Status 404 returned error can't find the container with id e36876a97fb3a6518d03c7ef576d921f199234b09e245ea6e63982d104e4aa9d Jul 23 11:33:21.227824 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.227796 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8bfk\" (UniqueName: \"kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk\") pod \"7ba7176a-ca26-441a-9152-d6433b77d692\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " Jul 23 11:33:21.227977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.227960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggkl4\" (UniqueName: \"kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4\") pod \"2d4667d3-0e1a-4271-9a83-8f714bcc6b26\" (UID: \"2d4667d3-0e1a-4271-9a83-8f714bcc6b26\") " Jul 23 11:33:21.228037 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.227993 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume\") pod \"7ba7176a-ca26-441a-9152-d6433b77d692\" (UID: \"7ba7176a-ca26-441a-9152-d6433b77d692\") " Jul 23 11:33:21.228876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.228837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7ba7176a-ca26-441a-9152-d6433b77d692" (UID: "7ba7176a-ca26-441a-9152-d6433b77d692"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:33:21.230385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.230361 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4" (OuterVolumeSpecName: "kube-api-access-ggkl4") pod "2d4667d3-0e1a-4271-9a83-8f714bcc6b26" (UID: "2d4667d3-0e1a-4271-9a83-8f714bcc6b26"). InnerVolumeSpecName "kube-api-access-ggkl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:33:21.230476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.230440 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk" (OuterVolumeSpecName: "kube-api-access-k8bfk") pod "7ba7176a-ca26-441a-9152-d6433b77d692" (UID: "7ba7176a-ca26-441a-9152-d6433b77d692"). InnerVolumeSpecName "kube-api-access-k8bfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:33:21.329031 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.329000 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkl4\" (UniqueName: \"kubernetes.io/projected/2d4667d3-0e1a-4271-9a83-8f714bcc6b26-kube-api-access-ggkl4\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:21.329031 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.329028 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ba7176a-ca26-441a-9152-d6433b77d692-extensions-socket-volume\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:21.329223 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.329040 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bfk\" (UniqueName: \"kubernetes.io/projected/7ba7176a-ca26-441a-9152-d6433b77d692-kube-api-access-k8bfk\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:21.625870 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.625827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" event={"ID":"9a3b996e-4708-4a70-ba2f-f5567cd666a8","Type":"ContainerStarted","Data":"b86e1d7d7bc3e4122cea7603a7a105d870e60de3798589e77fa7cb38308fe270"} Jul 23 11:33:21.627471 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.627440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d8d5c76b-cq2z2" event={"ID":"c9761335-16b5-4a27-8f71-1d18e0c7d871","Type":"ContainerStarted","Data":"09d846f439309c37dd9cddc38b1260cc06b7d999fc3181066092fde45c698463"} Jul 23 11:33:21.627594 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.627488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d8d5c76b-cq2z2" event={"ID":"c9761335-16b5-4a27-8f71-1d18e0c7d871","Type":"ContainerStarted","Data":"8407ce762620d0046728f5528e5ffb4a4eb47ca7a14ef7b854378beec9bc0228"} Jul 23 11:33:21.628965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.628944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" event={"ID":"a606d3b3-4b05-4ec9-ba75-eefd0736f411","Type":"ContainerStarted","Data":"4bf5ddcb6736a6d2e9c0f4db05884bf2937a568f386ec1deceee4443987091de"} Jul 23 11:33:21.629069 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.628993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:21.629069 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.629018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" event={"ID":"a606d3b3-4b05-4ec9-ba75-eefd0736f411","Type":"ContainerStarted","Data":"e36876a97fb3a6518d03c7ef576d921f199234b09e245ea6e63982d104e4aa9d"} Jul 23 11:33:21.630257 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.630221 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.630384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.630304 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" containerID="91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20" exitCode=0 Jul 23 11:33:21.630384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.630350 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.630485 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.630398 2564 scope.go:117] "RemoveContainer" containerID="91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20" Jul 23 11:33:21.632548 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.632527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" event={"ID":"0466c8b5-9683-471f-bcad-8c14c0c3d1a7","Type":"ContainerStarted","Data":"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02"} Jul 23 11:33:21.632662 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.632564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" event={"ID":"0466c8b5-9683-471f-bcad-8c14c0c3d1a7","Type":"ContainerStarted","Data":"b447bb6f727ce856e98bc5749b99d1b3a1a8f0ceefe0f17b882f8172d1572993"} Jul 23 11:33:21.632662 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.632606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:21.634007 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.633990 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ba7176a-ca26-441a-9152-d6433b77d692" containerID="26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0" exitCode=0 Jul 23 11:33:21.634142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.634030 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:21.641368 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.641353 2564 scope.go:117] "RemoveContainer" containerID="91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20" Jul 23 11:33:21.641665 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.641642 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20\": container with ID starting with 91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20 not found: ID does not exist" containerID="91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20" Jul 23 11:33:21.641783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.641674 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20"} err="failed to get container status \"91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20\": rpc error: code = NotFound desc = could not find container \"91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20\": container with ID starting with 91a883282a64202cae2731a75b0556547a719bbdc232e787d103e4a366bbba20 not found: ID does not exist" Jul 23 11:33:21.641783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.641692 2564 scope.go:117] "RemoveContainer" containerID="26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0" Jul 23 11:33:21.648338 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.648295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9sqzk" podStartSLOduration=1.260384454 podStartE2EDuration="22.648280922s" totalImagesPullingTime="21.387896468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:33:21.647853947 +0000 UTC m=+395.629851453" watchObservedRunningTime="2026-07-23 11:33:21.648280922 +0000 UTC m=+395.630278424" Jul 23 11:33:21.650669 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.650651 2564 scope.go:117] "RemoveContainer" containerID="26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0" Jul 23 11:33:21.650850 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.650824 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:21.650958 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.650941 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0\": container with ID starting with 26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0 not found: ID does not exist" containerID="26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0" Jul 23 11:33:21.651005 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.650965 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0"} err="failed to get container status \"26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0\": rpc error: code = NotFound desc = could not find container \"26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0\": container with ID starting with 26310eefd3b1b9ebf21e2ba3133f03be12b3b8e95264f06998da93cfdc671be0 not found: ID does not exist" Jul 23 11:33:21.652734 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.652715 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fb6v2\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fb6v2" Jul 23 11:33:21.692068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.692014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" podStartSLOduration=12.691999719 podStartE2EDuration="12.691999719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:33:21.690224086 +0000 UTC m=+395.672221592" watchObservedRunningTime="2026-07-23 11:33:21.691999719 +0000 UTC m=+395.673997222" Jul 23 11:33:21.692242 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:21.692152 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fzf5l\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fzf5l" Jul 23 11:33:21.692621 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.692596 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" podStartSLOduration=12.692589009 podStartE2EDuration="12.692589009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:33:21.673422014 +0000 UTC m=+395.655419517" watchObservedRunningTime="2026-07-23 11:33:21.692589009 +0000 UTC m=+395.674586512" Jul 23 11:33:21.714579 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.714533 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d8d5c76b-cq2z2" podStartSLOduration=7.71451868 podStartE2EDuration="7.71451868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:33:21.712010537 +0000 UTC m=+395.694008040" watchObservedRunningTime="2026-07-23 11:33:21.71451868 +0000 UTC m=+395.696516183" Jul 23 11:33:21.778728 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.778694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:21.832850 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:21.832819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:22.543447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:22.543413 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4667d3-0e1a-4271-9a83-8f714bcc6b26" path="/var/lib/kubelet/pods/2d4667d3-0e1a-4271-9a83-8f714bcc6b26/volumes" Jul 23 11:33:22.543875 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:22.543800 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba7176a-ca26-441a-9152-d6433b77d692" path="/var/lib/kubelet/pods/7ba7176a-ca26-441a-9152-d6433b77d692/volumes" Jul 23 11:33:24.395165 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.395132 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:24.395622 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.395316 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" podUID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" containerName="manager" containerID="cri-o://213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02" gracePeriod=10 Jul 23 11:33:24.641699 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.641670 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:24.653849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.653784 2564 generic.go:356] "Generic (PLEG): container finished" podID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" containerID="213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02" exitCode=0 Jul 23 11:33:24.653849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.653836 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" Jul 23 11:33:24.654034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.653862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" event={"ID":"0466c8b5-9683-471f-bcad-8c14c0c3d1a7","Type":"ContainerDied","Data":"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02"} Jul 23 11:33:24.654034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.653929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l" event={"ID":"0466c8b5-9683-471f-bcad-8c14c0c3d1a7","Type":"ContainerDied","Data":"b447bb6f727ce856e98bc5749b99d1b3a1a8f0ceefe0f17b882f8172d1572993"} Jul 23 11:33:24.654034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.653959 2564 scope.go:117] "RemoveContainer" containerID="213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02" Jul 23 11:33:24.663184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.663165 2564 scope.go:117] "RemoveContainer" containerID="213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02" Jul 23 11:33:24.663455 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:24.663423 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02\": container with ID starting with 213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02 not found: ID does not exist" containerID="213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02" Jul 23 11:33:24.663500 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.663457 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02"} err="failed to get container status \"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02\": rpc error: code = NotFound desc = could not find container \"213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02\": container with ID starting with 213610bdb64b1311dace06cfa9817b91e2262075156624275001194df8404b02 not found: ID does not exist" Jul 23 11:33:24.757114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.757074 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:33:24.757712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.757695 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" containerName="manager" Jul 23 11:33:24.757797 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.757717 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" containerName="manager" Jul 23 11:33:24.757865 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.757852 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" containerName="manager" Jul 23 11:33:24.759609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.759589 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume\") pod \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " Jul 23 11:33:24.759665 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.759617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gftrr\" (UniqueName: \"kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr\") pod \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\" (UID: \"0466c8b5-9683-471f-bcad-8c14c0c3d1a7\") " Jul 23 11:33:24.759949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.759928 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0466c8b5-9683-471f-bcad-8c14c0c3d1a7" (UID: "0466c8b5-9683-471f-bcad-8c14c0c3d1a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:33:24.761967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.761942 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr" (OuterVolumeSpecName: "kube-api-access-gftrr") pod "0466c8b5-9683-471f-bcad-8c14c0c3d1a7" (UID: "0466c8b5-9683-471f-bcad-8c14c0c3d1a7"). InnerVolumeSpecName "kube-api-access-gftrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:33:24.762879 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.762854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.777160 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.777135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:33:24.861200 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.861161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbddp\" (UniqueName: \"kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.861200 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.861201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.861417 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.861356 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-extensions-socket-volume\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:24.861417 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.861382 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gftrr\" (UniqueName: \"kubernetes.io/projected/0466c8b5-9683-471f-bcad-8c14c0c3d1a7-kube-api-access-gftrr\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:24.962188 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.962082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbddp\" (UniqueName: \"kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.962188 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.962125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.962625 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.962603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.971461 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.971419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbddp\" (UniqueName: \"kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrsn6\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:24.983538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.983506 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:24.988479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:24.988451 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4nq9l"] Jul 23 11:33:25.081064 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.081025 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:25.216118 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.216094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:33:25.217616 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:25.217590 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f78146_0176_4535_820e_cbef8a278975.slice/crio-f398dce414a273cd02e1b3d9e1fa058a9d1e2c2add557f1a7236a169264250c0 WatchSource:0}: Error finding container f398dce414a273cd02e1b3d9e1fa058a9d1e2c2add557f1a7236a169264250c0: Status 404 returned error can't find the container with id f398dce414a273cd02e1b3d9e1fa058a9d1e2c2add557f1a7236a169264250c0 Jul 23 11:33:25.353429 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.353384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:25.353601 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.353489 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:25.358447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.358424 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:25.662735 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.662700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" event={"ID":"97f78146-0176-4535-820e-cbef8a278975","Type":"ContainerStarted","Data":"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af"} Jul 23 11:33:25.663230 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.662783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:25.663230 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.662803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" event={"ID":"97f78146-0176-4535-820e-cbef8a278975","Type":"ContainerStarted","Data":"f398dce414a273cd02e1b3d9e1fa058a9d1e2c2add557f1a7236a169264250c0"} Jul 23 11:33:25.667943 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.667920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d8d5c76b-cq2z2" Jul 23 11:33:25.684917 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.684859 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" podStartSLOduration=1.6848429999999999 podStartE2EDuration="1.684843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:33:25.682947354 +0000 UTC m=+399.664944862" watchObservedRunningTime="2026-07-23 11:33:25.684843 +0000 UTC m=+399.666840502" Jul 23 11:33:25.732579 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.732547 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:33:25.861131 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.861093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:25.964198 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:25.964096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:26.063748 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:26.063716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:26.543905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:26.543875 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0466c8b5-9683-471f-bcad-8c14c0c3d1a7" path="/var/lib/kubelet/pods/0466c8b5-9683-471f-bcad-8c14c0c3d1a7/volumes" Jul 23 11:33:28.412284 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:28.412250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:28.509821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:28.509792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:31.274694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:31.274648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:32.644297 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:32.644265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbx5k" Jul 23 11:33:33.181601 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:33.181565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:33.283257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:33.283211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:35.767203 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:35.767166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:35.872513 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:35.872464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:36.671778 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:36.671741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:33:39.574081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:39.574046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:39.630710 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:39.630666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:40.936523 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:40.936493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:41.061171 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:41.061138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:47.786004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:47.785965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:47.883099 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:47.883045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:50.761747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:50.761712 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65885c47b6-hxxr4" podUID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" containerName="console" containerID="cri-o://d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920" gracePeriod=15 Jul 23 11:33:51.031218 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.031146 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:33:51.099323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099412 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099812 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099570 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpfv4\" (UniqueName: \"kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4\") pod \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\" (UID: \"54d17cf9-c9a7-4c42-a891-a8604b29e8c0\") " Jul 23 11:33:51.099890 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099864 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config" (OuterVolumeSpecName: "console-config") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:33:51.099950 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099902 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:33:51.100007 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099989 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-trusted-ca-bundle\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.100007 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:33:51.100117 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.100010 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.100117 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.099988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca" (OuterVolumeSpecName: "service-ca") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:33:51.102035 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.102000 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4" (OuterVolumeSpecName: "kube-api-access-fpfv4") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "kube-api-access-fpfv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:33:51.102177 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.102100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:33:51.102177 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.102151 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "54d17cf9-c9a7-4c42-a891-a8604b29e8c0" (UID: "54d17cf9-c9a7-4c42-a891-a8604b29e8c0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:33:51.201385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.201348 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpfv4\" (UniqueName: \"kubernetes.io/projected/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-kube-api-access-fpfv4\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.201385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.201377 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.201385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.201388 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.201604 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.201399 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-console-oauth-config\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.201604 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.201409 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54d17cf9-c9a7-4c42-a891-a8604b29e8c0-oauth-serving-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:33:51.816314 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.816277 2564 generic.go:356] "Generic (PLEG): container finished" podID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" containerID="d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920" exitCode=0 Jul 23 11:33:51.816807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.816325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65885c47b6-hxxr4" Jul 23 11:33:51.816807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.816357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65885c47b6-hxxr4" event={"ID":"54d17cf9-c9a7-4c42-a891-a8604b29e8c0","Type":"ContainerDied","Data":"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920"} Jul 23 11:33:51.816807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.816407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65885c47b6-hxxr4" event={"ID":"54d17cf9-c9a7-4c42-a891-a8604b29e8c0","Type":"ContainerDied","Data":"23f6a2beef5865eb477f58298c9929016e9e3cbdfaa17d2e993281d8cd3fb68a"} Jul 23 11:33:51.816807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.816426 2564 scope.go:117] "RemoveContainer" containerID="d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920" Jul 23 11:33:51.826376 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.826358 2564 scope.go:117] "RemoveContainer" containerID="d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920" Jul 23 11:33:51.826643 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:33:51.826628 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920\": container with ID starting with d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920 not found: ID does not exist" containerID="d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920" Jul 23 11:33:51.826700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.826650 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920"} err="failed to get container status \"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920\": rpc error: code = NotFound desc = could not find container \"d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920\": container with ID starting with d8ef23a30008922d298e08529c886b31b42878c7f2bf19a786443ffb81c00920 not found: ID does not exist" Jul 23 11:33:51.838120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.838092 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:33:51.841854 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:51.841827 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65885c47b6-hxxr4"] Jul 23 11:33:52.543203 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:52.543170 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" path="/var/lib/kubelet/pods/54d17cf9-c9a7-4c42-a891-a8604b29e8c0/volumes" Jul 23 11:33:52.595377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:52.595339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:52.695566 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:52.695532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:55.609807 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.609775 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:33:55.610257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.610132 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" containerName="console" Jul 23 11:33:55.610257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.610144 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" containerName="console" Jul 23 11:33:55.610383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.610262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d17cf9-c9a7-4c42-a891-a8604b29e8c0" containerName="console" Jul 23 11:33:55.622440 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.622412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.623055 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.623033 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:33:55.624728 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.624695 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 11:33:55.706420 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.706390 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:33:55.740477 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.740445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznhr\" (UniqueName: \"kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.740631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.740503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.841280 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.841252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznhr\" (UniqueName: \"kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.841479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.841309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.841896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.841879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.849436 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.849407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:55.849650 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.849632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznhr\" (UniqueName: \"kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr\") pod \"limitador-limitador-64c94fd64f-hsw7c\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:55.910311 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.910225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:55.936738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:55.936711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:33:56.081681 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.081652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:33:56.083643 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:56.083611 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f6f68c_d0b2_4bcb_a4a2_8aad23ebcb1a.slice/crio-16d8c116ed0173f3e1b8f99439fedfd3fc27aa9ed93c4d63e3d66871329072bb WatchSource:0}: Error finding container 16d8c116ed0173f3e1b8f99439fedfd3fc27aa9ed93c4d63e3d66871329072bb: Status 404 returned error can't find the container with id 16d8c116ed0173f3e1b8f99439fedfd3fc27aa9ed93c4d63e3d66871329072bb Jul 23 11:33:56.455120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.455085 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:33:56.459955 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.459937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:33:56.462420 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.462400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-45nfw\"" Jul 23 11:33:56.467641 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.467592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:33:56.478281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.478259 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:33:56.481926 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.481907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:33:56.490068 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.490047 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:33:56.501393 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.501369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:56.547538 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.547508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7krj\" (UniqueName: \"kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj\") pod \"authorino-75d556bd77-6pw4v\" (UID: \"3a82a800-838c-45d5-861a-1d098610c5ba\") " pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:33:56.547696 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.547592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwtxx\" (UniqueName: \"kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx\") pod \"authorino-58cd955675-h4fzm\" (UID: \"25e54bb3-55ab-40ec-a196-4e6305a998b4\") " pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:33:56.597488 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.597458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:33:56.648850 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.648818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7krj\" (UniqueName: \"kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj\") pod \"authorino-75d556bd77-6pw4v\" (UID: \"3a82a800-838c-45d5-861a-1d098610c5ba\") " pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:33:56.649219 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.648872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwtxx\" (UniqueName: \"kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx\") pod \"authorino-58cd955675-h4fzm\" (UID: \"25e54bb3-55ab-40ec-a196-4e6305a998b4\") " pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:33:56.657634 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.657605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7krj\" (UniqueName: \"kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj\") pod \"authorino-75d556bd77-6pw4v\" (UID: \"3a82a800-838c-45d5-861a-1d098610c5ba\") " pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:33:56.657634 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.657617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwtxx\" (UniqueName: \"kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx\") pod \"authorino-58cd955675-h4fzm\" (UID: \"25e54bb3-55ab-40ec-a196-4e6305a998b4\") " pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:33:56.772317 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.771703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:33:56.791249 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.791221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:33:56.855916 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.855016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" event={"ID":"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a","Type":"ContainerStarted","Data":"16d8c116ed0173f3e1b8f99439fedfd3fc27aa9ed93c4d63e3d66871329072bb"} Jul 23 11:33:56.943388 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.942992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:33:56.973527 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:56.973496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:33:56.981778 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:33:56.981728 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a82a800_838c_45d5_861a_1d098610c5ba.slice/crio-df4936ec5104d65fd5b58b31fa167a8c8a086641b875617ef12eab8bdc120399 WatchSource:0}: Error finding container df4936ec5104d65fd5b58b31fa167a8c8a086641b875617ef12eab8bdc120399: Status 404 returned error can't find the container with id df4936ec5104d65fd5b58b31fa167a8c8a086641b875617ef12eab8bdc120399 Jul 23 11:33:57.872805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:57.871022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6pw4v" event={"ID":"3a82a800-838c-45d5-861a-1d098610c5ba","Type":"ContainerStarted","Data":"df4936ec5104d65fd5b58b31fa167a8c8a086641b875617ef12eab8bdc120399"} Jul 23 11:33:57.876108 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:33:57.875510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h4fzm" event={"ID":"25e54bb3-55ab-40ec-a196-4e6305a998b4","Type":"ContainerStarted","Data":"ac66a21d157253fac6941013ed6e381ccd4f555b3e463be9c20a5e61fa279f1f"} Jul 23 11:34:01.902919 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.902876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" event={"ID":"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a","Type":"ContainerStarted","Data":"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96"} Jul 23 11:34:01.903392 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.903067 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:34:01.904480 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.904454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6pw4v" event={"ID":"3a82a800-838c-45d5-861a-1d098610c5ba","Type":"ContainerStarted","Data":"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7"} Jul 23 11:34:01.905973 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.905942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h4fzm" event={"ID":"25e54bb3-55ab-40ec-a196-4e6305a998b4","Type":"ContainerStarted","Data":"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115"} Jul 23 11:34:01.922545 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.922495 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" podStartSLOduration=1.6549088140000001 podStartE2EDuration="6.922481485s" totalImagesPullingTime="5.267572671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:01.920141169 +0000 UTC m=+435.902138673" watchObservedRunningTime="2026-07-23 11:34:01.922481485 +0000 UTC m=+435.904478989" Jul 23 11:34:01.933487 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.933445 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-6pw4v" podStartSLOduration=1.552108329 podStartE2EDuration="5.933427087s" totalImagesPullingTime="4.381318758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:01.933077334 +0000 UTC m=+435.915074850" watchObservedRunningTime="2026-07-23 11:34:01.933427087 +0000 UTC m=+435.915424591" Jul 23 11:34:01.949435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.949387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-h4fzm" podStartSLOduration=1.543799667 podStartE2EDuration="5.949369738s" totalImagesPullingTime="4.405570071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:01.947178206 +0000 UTC m=+435.929175709" watchObservedRunningTime="2026-07-23 11:34:01.949369738 +0000 UTC m=+435.931367616" Jul 23 11:34:01.976363 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:01.976331 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:34:03.917129 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:03.917088 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-h4fzm" podUID="25e54bb3-55ab-40ec-a196-4e6305a998b4" containerName="authorino" containerID="cri-o://742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115" gracePeriod=30 Jul 23 11:34:04.158713 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.158686 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:34:04.222054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.222032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwtxx\" (UniqueName: \"kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx\") pod \"25e54bb3-55ab-40ec-a196-4e6305a998b4\" (UID: \"25e54bb3-55ab-40ec-a196-4e6305a998b4\") " Jul 23 11:34:04.224125 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.224098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx" (OuterVolumeSpecName: "kube-api-access-bwtxx") pod "25e54bb3-55ab-40ec-a196-4e6305a998b4" (UID: "25e54bb3-55ab-40ec-a196-4e6305a998b4"). InnerVolumeSpecName "kube-api-access-bwtxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:34:04.323278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.323252 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwtxx\" (UniqueName: \"kubernetes.io/projected/25e54bb3-55ab-40ec-a196-4e6305a998b4-kube-api-access-bwtxx\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:04.923712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.923610 2564 generic.go:356] "Generic (PLEG): container finished" podID="25e54bb3-55ab-40ec-a196-4e6305a998b4" containerID="742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115" exitCode=0 Jul 23 11:34:04.923712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.923655 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-h4fzm" Jul 23 11:34:04.923712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.923652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h4fzm" event={"ID":"25e54bb3-55ab-40ec-a196-4e6305a998b4","Type":"ContainerDied","Data":"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115"} Jul 23 11:34:04.923712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.923697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-h4fzm" event={"ID":"25e54bb3-55ab-40ec-a196-4e6305a998b4","Type":"ContainerDied","Data":"ac66a21d157253fac6941013ed6e381ccd4f555b3e463be9c20a5e61fa279f1f"} Jul 23 11:34:04.923712 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.923714 2564 scope.go:117] "RemoveContainer" containerID="742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115" Jul 23 11:34:04.932912 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.932894 2564 scope.go:117] "RemoveContainer" containerID="742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115" Jul 23 11:34:04.933180 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:34:04.933162 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115\": container with ID starting with 742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115 not found: ID does not exist" containerID="742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115" Jul 23 11:34:04.933226 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.933193 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115"} err="failed to get container status \"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115\": rpc error: code = NotFound desc = could not find container \"742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115\": container with ID starting with 742d56dfa55b3e6f02574c952ce2a14c73c5aa88c710fd1366e5bc327fc59115 not found: ID does not exist" Jul 23 11:34:04.940435 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.940411 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:34:04.942722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:04.942699 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-h4fzm"] Jul 23 11:34:06.542684 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:06.542611 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e54bb3-55ab-40ec-a196-4e6305a998b4" path="/var/lib/kubelet/pods/25e54bb3-55ab-40ec-a196-4e6305a998b4/volumes" Jul 23 11:34:10.601060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:10.601012 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:34:10.601558 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:10.601261 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" podUID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" containerName="limitador" containerID="cri-o://d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96" gracePeriod=30 Jul 23 11:34:10.603700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:10.603680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:34:11.156786 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.156742 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:34:11.279710 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.279680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file\") pod \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " Jul 23 11:34:11.279930 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.279756 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vznhr\" (UniqueName: \"kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr\") pod \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\" (UID: \"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a\") " Jul 23 11:34:11.280123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.280100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file" (OuterVolumeSpecName: "config-file") pod "27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" (UID: "27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:34:11.282006 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.281982 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr" (OuterVolumeSpecName: "kube-api-access-vznhr") pod "27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" (UID: "27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a"). InnerVolumeSpecName "kube-api-access-vznhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:34:11.380794 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.380730 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-config-file\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:11.380949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.380817 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznhr\" (UniqueName: \"kubernetes.io/projected/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a-kube-api-access-vznhr\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:11.961823 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.961787 2564 generic.go:356] "Generic (PLEG): container finished" podID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" containerID="d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96" exitCode=0 Jul 23 11:34:11.962257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.961830 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" Jul 23 11:34:11.962257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.961865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" event={"ID":"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a","Type":"ContainerDied","Data":"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96"} Jul 23 11:34:11.962257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.961916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hsw7c" event={"ID":"27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a","Type":"ContainerDied","Data":"16d8c116ed0173f3e1b8f99439fedfd3fc27aa9ed93c4d63e3d66871329072bb"} Jul 23 11:34:11.962257 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.961938 2564 scope.go:117] "RemoveContainer" containerID="d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96" Jul 23 11:34:11.971394 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.971371 2564 scope.go:117] "RemoveContainer" containerID="d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96" Jul 23 11:34:11.971674 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:34:11.971655 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96\": container with ID starting with d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96 not found: ID does not exist" containerID="d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96" Jul 23 11:34:11.971741 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.971687 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96"} err="failed to get container status \"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96\": rpc error: code = NotFound desc = could not find container \"d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96\": container with ID starting with d662bd1e2bdfd4923529ae27bff51acd05b6d53ae3ef28376201bd63e3e57d96 not found: ID does not exist" Jul 23 11:34:11.983129 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.983107 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:34:11.987350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:11.987331 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hsw7c"] Jul 23 11:34:12.543328 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:12.543292 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" path="/var/lib/kubelet/pods/27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a/volumes" Jul 23 11:34:14.039668 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.039634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:14.139175 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.139143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:14.491234 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491206 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz"] Jul 23 11:34:14.491590 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491576 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" containerName="limitador" Jul 23 11:34:14.491639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491591 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" containerName="limitador" Jul 23 11:34:14.491639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491602 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e54bb3-55ab-40ec-a196-4e6305a998b4" containerName="authorino" Jul 23 11:34:14.491639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491608 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e54bb3-55ab-40ec-a196-4e6305a998b4" containerName="authorino" Jul 23 11:34:14.491733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491671 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e54bb3-55ab-40ec-a196-4e6305a998b4" containerName="authorino" Jul 23 11:34:14.491733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.491681 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f6f68c-d0b2-4bcb-a4a2-8aad23ebcb1a" containerName="limitador" Jul 23 11:34:14.496016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.495997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.498584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.498564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 11:34:14.498697 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.498655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 11:34:14.499644 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.499624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 11:34:14.500008 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.499637 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dngss\"" Jul 23 11:34:14.503400 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.503381 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz"] Jul 23 11:34:14.529126 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.529096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:14.609071 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.609031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gd25\" (UniqueName: \"kubernetes.io/projected/54288d3e-132d-4e08-b035-75091ff814ad-kube-api-access-8gd25\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.609251 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.609083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54288d3e-132d-4e08-b035-75091ff814ad-data\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.710510 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.710475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gd25\" (UniqueName: \"kubernetes.io/projected/54288d3e-132d-4e08-b035-75091ff814ad-kube-api-access-8gd25\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.710510 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.710515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54288d3e-132d-4e08-b035-75091ff814ad-data\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.710881 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.710864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/54288d3e-132d-4e08-b035-75091ff814ad-data\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.719146 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.719113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gd25\" (UniqueName: \"kubernetes.io/projected/54288d3e-132d-4e08-b035-75091ff814ad-kube-api-access-8gd25\") pod \"postgres-69fb4594cf-ws8rz\" (UID: \"54288d3e-132d-4e08-b035-75091ff814ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.808324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.808228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:14.943861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.943837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz"] Jul 23 11:34:14.945478 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:34:14.945448 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54288d3e_132d_4e08_b035_75091ff814ad.slice/crio-0c8f4b8a76ad270c781a94ad71d21c551fff71fff973ab8e549d4bfb25c95a64 WatchSource:0}: Error finding container 0c8f4b8a76ad270c781a94ad71d21c551fff71fff973ab8e549d4bfb25c95a64: Status 404 returned error can't find the container with id 0c8f4b8a76ad270c781a94ad71d21c551fff71fff973ab8e549d4bfb25c95a64 Jul 23 11:34:14.980546 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:14.980516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" event={"ID":"54288d3e-132d-4e08-b035-75091ff814ad","Type":"ContainerStarted","Data":"0c8f4b8a76ad270c781a94ad71d21c551fff71fff973ab8e549d4bfb25c95a64"} Jul 23 11:34:20.022078 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:20.021976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" event={"ID":"54288d3e-132d-4e08-b035-75091ff814ad","Type":"ContainerStarted","Data":"b8698a3eb06cdb22e98642bd70db44207b0b2436dfe74ee162f353b887be1bf6"} Jul 23 11:34:20.022078 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:20.022056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:20.040370 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:20.040315 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" podStartSLOduration=1.229923038 podStartE2EDuration="6.040297316s" totalImagesPullingTime="4.810374278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:20.038408012 +0000 UTC m=+454.020405515" watchObservedRunningTime="2026-07-23 11:34:20.040297316 +0000 UTC m=+454.022294874" Jul 23 11:34:26.057299 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.057270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws8rz" Jul 23 11:34:26.366411 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.366326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:26.581054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.581014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:26.598911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.598879 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:26.604445 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.604422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:26.607861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.607839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:26.617219 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.617139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7w7\" (UniqueName: \"kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7\") pod \"authorino-5d4b67c446-ljmc5\" (UID: \"232250db-d9bf-47fc-8928-8f7dc0b3ebca\") " pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:26.718148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.718109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7w7\" (UniqueName: \"kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7\") pod \"authorino-5d4b67c446-ljmc5\" (UID: \"232250db-d9bf-47fc-8928-8f7dc0b3ebca\") " pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:26.726165 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.726139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7w7\" (UniqueName: \"kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7\") pod \"authorino-5d4b67c446-ljmc5\" (UID: \"232250db-d9bf-47fc-8928-8f7dc0b3ebca\") " pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:26.828114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.828078 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:26.828334 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.828314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:26.850748 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.850722 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cff545fc4-plkq6"] Jul 23 11:34:26.855682 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.855659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:26.860833 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.860810 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cff545fc4-plkq6"] Jul 23 11:34:26.919539 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.919511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdgfp\" (UniqueName: \"kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp\") pod \"authorino-cff545fc4-plkq6\" (UID: \"f7339bd7-64fc-4d08-a9fb-f0d68235944a\") " pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:26.961464 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.961430 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cff545fc4-plkq6"] Jul 23 11:34:26.961801 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:34:26.961756 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rdgfp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-cff545fc4-plkq6" podUID="f7339bd7-64fc-4d08-a9fb-f0d68235944a" Jul 23 11:34:26.964717 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.964695 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:26.966678 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:34:26.966649 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod232250db_d9bf_47fc_8928_8f7dc0b3ebca.slice/crio-bb06a310a0cc91c225cf5dc913e54a102c35ae280b624b6d07a301a53fcef2e0 WatchSource:0}: Error finding container bb06a310a0cc91c225cf5dc913e54a102c35ae280b624b6d07a301a53fcef2e0: Status 404 returned error can't find the container with id bb06a310a0cc91c225cf5dc913e54a102c35ae280b624b6d07a301a53fcef2e0 Jul 23 11:34:26.987262 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.987230 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:34:26.992592 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.992571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:26.995375 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.995355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 11:34:26.997498 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:26.997447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:34:27.020050 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.020020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.020177 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.020060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgfp\" (UniqueName: \"kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp\") pod \"authorino-cff545fc4-plkq6\" (UID: \"f7339bd7-64fc-4d08-a9fb-f0d68235944a\") " pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:27.020177 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.020126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgn7h\" (UniqueName: \"kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.030888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.030868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdgfp\" (UniqueName: \"kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp\") pod \"authorino-cff545fc4-plkq6\" (UID: \"f7339bd7-64fc-4d08-a9fb-f0d68235944a\") " pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:27.063370 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.063339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" event={"ID":"232250db-d9bf-47fc-8928-8f7dc0b3ebca","Type":"ContainerStarted","Data":"bb06a310a0cc91c225cf5dc913e54a102c35ae280b624b6d07a301a53fcef2e0"} Jul 23 11:34:27.063861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.063386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:27.070803 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.070783 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:27.120578 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.120541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdgfp\" (UniqueName: \"kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp\") pod \"f7339bd7-64fc-4d08-a9fb-f0d68235944a\" (UID: \"f7339bd7-64fc-4d08-a9fb-f0d68235944a\") " Jul 23 11:34:27.120788 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.120674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.120788 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.120721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgn7h\" (UniqueName: \"kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.122851 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.122814 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp" (OuterVolumeSpecName: "kube-api-access-rdgfp") pod "f7339bd7-64fc-4d08-a9fb-f0d68235944a" (UID: "f7339bd7-64fc-4d08-a9fb-f0d68235944a"). InnerVolumeSpecName "kube-api-access-rdgfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:34:27.123266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.123250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.128971 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.128947 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgn7h\" (UniqueName: \"kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h\") pod \"authorino-cbc8f5f54-cl2pz\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.221433 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.221400 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdgfp\" (UniqueName: \"kubernetes.io/projected/f7339bd7-64fc-4d08-a9fb-f0d68235944a-kube-api-access-rdgfp\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:27.303360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.303324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:34:27.441304 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:27.441282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:34:27.443054 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:34:27.443034 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69c8f731_8ab5_4ab0_b6e2_5b2b2fe50631.slice/crio-f32064ccc9728f8b91c24696d65e0562f88cbc17349dbbe9b30cd50c581c7dee WatchSource:0}: Error finding container f32064ccc9728f8b91c24696d65e0562f88cbc17349dbbe9b30cd50c581c7dee: Status 404 returned error can't find the container with id f32064ccc9728f8b91c24696d65e0562f88cbc17349dbbe9b30cd50c581c7dee Jul 23 11:34:28.076578 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.076461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" event={"ID":"232250db-d9bf-47fc-8928-8f7dc0b3ebca","Type":"ContainerStarted","Data":"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9"} Jul 23 11:34:28.076578 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.076544 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" podUID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" containerName="authorino" containerID="cri-o://4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9" gracePeriod=30 Jul 23 11:34:28.078485 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.078459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" event={"ID":"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631","Type":"ContainerStarted","Data":"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a"} Jul 23 11:34:28.078618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.078496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" event={"ID":"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631","Type":"ContainerStarted","Data":"f32064ccc9728f8b91c24696d65e0562f88cbc17349dbbe9b30cd50c581c7dee"} Jul 23 11:34:28.078618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.078516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cff545fc4-plkq6" Jul 23 11:34:28.093385 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.093319 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" podStartSLOduration=1.5376084570000002 podStartE2EDuration="2.093306299s" totalImagesPullingTime="555.697842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:28.09088256 +0000 UTC m=+462.072880066" watchObservedRunningTime="2026-07-23 11:34:28.093306299 +0000 UTC m=+462.075303802" Jul 23 11:34:28.107664 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.107620 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" podStartSLOduration=1.802515624 podStartE2EDuration="2.107605533s" totalImagesPullingTime="305.089909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:34:28.105934819 +0000 UTC m=+462.087932322" watchObservedRunningTime="2026-07-23 11:34:28.107605533 +0000 UTC m=+462.089603035" Jul 23 11:34:28.134450 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.134415 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:34:28.134652 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.134627 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-6pw4v" podUID="3a82a800-838c-45d5-861a-1d098610c5ba" containerName="authorino" containerID="cri-o://2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7" gracePeriod=30 Jul 23 11:34:28.176678 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.176647 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cff545fc4-plkq6"] Jul 23 11:34:28.180459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.180427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cff545fc4-plkq6"] Jul 23 11:34:28.369428 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.369405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:28.399679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.399654 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:34:28.433070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.433042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj7w7\" (UniqueName: \"kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7\") pod \"232250db-d9bf-47fc-8928-8f7dc0b3ebca\" (UID: \"232250db-d9bf-47fc-8928-8f7dc0b3ebca\") " Jul 23 11:34:28.433251 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.433103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7krj\" (UniqueName: \"kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj\") pod \"3a82a800-838c-45d5-861a-1d098610c5ba\" (UID: \"3a82a800-838c-45d5-861a-1d098610c5ba\") " Jul 23 11:34:28.435244 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.435213 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7" (OuterVolumeSpecName: "kube-api-access-jj7w7") pod "232250db-d9bf-47fc-8928-8f7dc0b3ebca" (UID: "232250db-d9bf-47fc-8928-8f7dc0b3ebca"). InnerVolumeSpecName "kube-api-access-jj7w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:34:28.435376 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.435263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj" (OuterVolumeSpecName: "kube-api-access-j7krj") pod "3a82a800-838c-45d5-861a-1d098610c5ba" (UID: "3a82a800-838c-45d5-861a-1d098610c5ba"). InnerVolumeSpecName "kube-api-access-j7krj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:34:28.533885 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.533852 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7w7\" (UniqueName: \"kubernetes.io/projected/232250db-d9bf-47fc-8928-8f7dc0b3ebca-kube-api-access-jj7w7\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:28.533885 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.533879 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7krj\" (UniqueName: \"kubernetes.io/projected/3a82a800-838c-45d5-861a-1d098610c5ba-kube-api-access-j7krj\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:34:28.543323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:28.543284 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7339bd7-64fc-4d08-a9fb-f0d68235944a" path="/var/lib/kubelet/pods/f7339bd7-64fc-4d08-a9fb-f0d68235944a/volumes" Jul 23 11:34:29.085571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.085481 2564 generic.go:356] "Generic (PLEG): container finished" podID="3a82a800-838c-45d5-861a-1d098610c5ba" containerID="2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7" exitCode=0 Jul 23 11:34:29.085571 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.085520 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6pw4v" Jul 23 11:34:29.086115 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.085573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6pw4v" event={"ID":"3a82a800-838c-45d5-861a-1d098610c5ba","Type":"ContainerDied","Data":"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7"} Jul 23 11:34:29.086115 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.085623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6pw4v" event={"ID":"3a82a800-838c-45d5-861a-1d098610c5ba","Type":"ContainerDied","Data":"df4936ec5104d65fd5b58b31fa167a8c8a086641b875617ef12eab8bdc120399"} Jul 23 11:34:29.086115 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.085646 2564 scope.go:117] "RemoveContainer" containerID="2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7" Jul 23 11:34:29.087070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.086960 2564 generic.go:356] "Generic (PLEG): container finished" podID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" containerID="4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9" exitCode=0 Jul 23 11:34:29.087070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.087000 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" Jul 23 11:34:29.087070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.087038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" event={"ID":"232250db-d9bf-47fc-8928-8f7dc0b3ebca","Type":"ContainerDied","Data":"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9"} Jul 23 11:34:29.087070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.087066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljmc5" event={"ID":"232250db-d9bf-47fc-8928-8f7dc0b3ebca","Type":"ContainerDied","Data":"bb06a310a0cc91c225cf5dc913e54a102c35ae280b624b6d07a301a53fcef2e0"} Jul 23 11:34:29.094983 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.094966 2564 scope.go:117] "RemoveContainer" containerID="2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7" Jul 23 11:34:29.095256 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:34:29.095235 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7\": container with ID starting with 2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7 not found: ID does not exist" containerID="2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7" Jul 23 11:34:29.095318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.095263 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7"} err="failed to get container status \"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7\": rpc error: code = NotFound desc = could not find container \"2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7\": container with ID starting with 2916aabd0d0b5f11223f1be8cba7bdd93a2f7e7901be07f825f289f8618126a7 not found: ID does not exist" Jul 23 11:34:29.095318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.095280 2564 scope.go:117] "RemoveContainer" containerID="4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9" Jul 23 11:34:29.103190 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.103174 2564 scope.go:117] "RemoveContainer" containerID="4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9" Jul 23 11:34:29.103444 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:34:29.103427 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9\": container with ID starting with 4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9 not found: ID does not exist" containerID="4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9" Jul 23 11:34:29.103497 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.103460 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9"} err="failed to get container status \"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9\": rpc error: code = NotFound desc = could not find container \"4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9\": container with ID starting with 4fad2ce6ace8900405d287f1b9d288da7f6fab507f79bb22a7e3052999cca1a9 not found: ID does not exist" Jul 23 11:34:29.106690 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.106672 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:34:29.110499 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.110480 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6pw4v"] Jul 23 11:34:29.121023 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.121001 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:29.124873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:29.124855 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljmc5"] Jul 23 11:34:30.542584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:30.542552 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" path="/var/lib/kubelet/pods/232250db-d9bf-47fc-8928-8f7dc0b3ebca/volumes" Jul 23 11:34:30.542955 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:30.542870 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a82a800-838c-45d5-861a-1d098610c5ba" path="/var/lib/kubelet/pods/3a82a800-838c-45d5-861a-1d098610c5ba/volumes" Jul 23 11:34:32.793785 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:32.793735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:32.894218 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:32.894192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:32.948866 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:32.948835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:33.082039 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:33.081959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:36.049425 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:36.049395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:36.449979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:36.449885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:34:36.550959 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:34:36.550926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:09.847122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:09.847066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:11.795709 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:11.795676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:11.893962 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:11.893918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:12.441859 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:12.441825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:12.868266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:12.868233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:12.995665 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:12.995633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:13.072246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072207 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xzrb4"] Jul 23 11:35:13.072631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072613 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" containerName="authorino" Jul 23 11:35:13.072631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072629 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" containerName="authorino" Jul 23 11:35:13.072815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072649 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a82a800-838c-45d5-861a-1d098610c5ba" containerName="authorino" Jul 23 11:35:13.072815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072654 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a82a800-838c-45d5-861a-1d098610c5ba" containerName="authorino" Jul 23 11:35:13.072815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072720 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a82a800-838c-45d5-861a-1d098610c5ba" containerName="authorino" Jul 23 11:35:13.072815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.072736 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="232250db-d9bf-47fc-8928-8f7dc0b3ebca" containerName="authorino" Jul 23 11:35:13.079744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.079719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.084975 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.084948 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kj8x8\"" Jul 23 11:35:13.084975 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.084968 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 11:35:13.087162 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.087138 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xzrb4"] Jul 23 11:35:13.129516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.129442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9msh\" (UniqueName: \"kubernetes.io/projected/4ecc94e6-3546-4d27-85dc-973a6742c957-kube-api-access-p9msh\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.129627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.129574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ecc94e6-3546-4d27-85dc-973a6742c957-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.158132 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.158108 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-b8z4d"] Jul 23 11:35:13.160582 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.160566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.171054 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.171027 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-b8z4d"] Jul 23 11:35:13.230315 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.230276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b34262-a94d-432d-a805-8178c5fa049e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.230481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.230324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5fp\" (UniqueName: \"kubernetes.io/projected/70b34262-a94d-432d-a805-8178c5fa049e-kube-api-access-fn5fp\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.230481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.230439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ecc94e6-3546-4d27-85dc-973a6742c957-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.230599 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.230495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9msh\" (UniqueName: \"kubernetes.io/projected/4ecc94e6-3546-4d27-85dc-973a6742c957-kube-api-access-p9msh\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.231195 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.231173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ecc94e6-3546-4d27-85dc-973a6742c957-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.239042 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.239019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9msh\" (UniqueName: \"kubernetes.io/projected/4ecc94e6-3546-4d27-85dc-973a6742c957-kube-api-access-p9msh\") pod \"payload-pre-processing-7677b774bb-xzrb4\" (UID: \"4ecc94e6-3546-4d27-85dc-973a6742c957\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.331294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.331268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b34262-a94d-432d-a805-8178c5fa049e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.331427 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.331301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5fp\" (UniqueName: \"kubernetes.io/projected/70b34262-a94d-432d-a805-8178c5fa049e-kube-api-access-fn5fp\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.331949 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.331930 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b34262-a94d-432d-a805-8178c5fa049e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.340150 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.340120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5fp\" (UniqueName: \"kubernetes.io/projected/70b34262-a94d-432d-a805-8178c5fa049e-kube-api-access-fn5fp\") pod \"payload-processing-5bf47cd988-b8z4d\" (UID: \"70b34262-a94d-432d-a805-8178c5fa049e\") " pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.392735 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.392681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:13.472035 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.472007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:13.541831 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.541717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xzrb4"] Jul 23 11:35:13.544108 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:35:13.544069 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecc94e6_3546_4d27_85dc_973a6742c957.slice/crio-04ccb366efa91fabf40a6a2a70b9995b326f9c3b168e4e4c6a7bb4aba7cd8b90 WatchSource:0}: Error finding container 04ccb366efa91fabf40a6a2a70b9995b326f9c3b168e4e4c6a7bb4aba7cd8b90: Status 404 returned error can't find the container with id 04ccb366efa91fabf40a6a2a70b9995b326f9c3b168e4e4c6a7bb4aba7cd8b90 Jul 23 11:35:13.614100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:13.614075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-b8z4d"] Jul 23 11:35:13.616910 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:35:13.616890 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b34262_a94d_432d_a805_8178c5fa049e.slice/crio-a2b32cec51bdc88be9d1d2422f701257a5d2871ca706024a577fabb7028b6a3b WatchSource:0}: Error finding container a2b32cec51bdc88be9d1d2422f701257a5d2871ca706024a577fabb7028b6a3b: Status 404 returned error can't find the container with id a2b32cec51bdc88be9d1d2422f701257a5d2871ca706024a577fabb7028b6a3b Jul 23 11:35:14.339594 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:14.339551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" event={"ID":"70b34262-a94d-432d-a805-8178c5fa049e","Type":"ContainerStarted","Data":"a2b32cec51bdc88be9d1d2422f701257a5d2871ca706024a577fabb7028b6a3b"} Jul 23 11:35:14.341164 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:14.341123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" event={"ID":"4ecc94e6-3546-4d27-85dc-973a6742c957","Type":"ContainerStarted","Data":"04ccb366efa91fabf40a6a2a70b9995b326f9c3b168e4e4c6a7bb4aba7cd8b90"} Jul 23 11:35:16.353519 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.353489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" event={"ID":"70b34262-a94d-432d-a805-8178c5fa049e","Type":"ContainerStarted","Data":"8dc352b771b7e6d2c95744cc8cbfd82cc50fd4252053ff7dcc7a85a8308091e0"} Jul 23 11:35:16.353988 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.353549 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:16.355092 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.355072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" event={"ID":"4ecc94e6-3546-4d27-85dc-973a6742c957","Type":"ContainerStarted","Data":"46a28a7038ea36827bec96fcabf593ed889b209c958c6973197d83f81a087d02"} Jul 23 11:35:16.355147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.355123 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:16.369733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.369689 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" podStartSLOduration=1.58861238 podStartE2EDuration="3.369674238s" totalImagesPullingTime="1.781061858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:16.368535097 +0000 UTC m=+510.350532602" watchObservedRunningTime="2026-07-23 11:35:16.369674238 +0000 UTC m=+510.351671793" Jul 23 11:35:16.382821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:16.382753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" podStartSLOduration=1.534386183 podStartE2EDuration="3.382722705s" totalImagesPullingTime="1.848336522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:16.382606429 +0000 UTC m=+510.364603934" watchObservedRunningTime="2026-07-23 11:35:16.382722705 +0000 UTC m=+510.364720200" Jul 23 11:35:21.735607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:21.735566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:22.936873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:22.936832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:23.032994 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:23.032956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:27.362812 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:27.362783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xzrb4" Jul 23 11:35:27.363236 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:27.362857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-b8z4d" Jul 23 11:35:32.666112 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:32.666079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:40.531508 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.531466 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf"] Jul 23 11:35:40.535183 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.535161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.537687 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.537663 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 11:35:40.538628 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.538600 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q2mvb\"" Jul 23 11:35:40.538738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.538683 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 23 11:35:40.538738 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.538684 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 11:35:40.549849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.549825 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf"] Jul 23 11:35:40.561225 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.561182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:40.676056 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.676242 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.676242 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c425ab8-0da3-4029-8406-1667e9262aeb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.676242 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24255\" (UniqueName: \"kubernetes.io/projected/6c425ab8-0da3-4029-8406-1667e9262aeb-kube-api-access-24255\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.676399 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.676399 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.676277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777423 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777423 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c425ab8-0da3-4029-8406-1667e9262aeb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24255\" (UniqueName: \"kubernetes.io/projected/6c425ab8-0da3-4029-8406-1667e9262aeb-kube-api-access-24255\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.777977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.777954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.778326 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.778229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.778326 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.778287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.779907 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.779888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c425ab8-0da3-4029-8406-1667e9262aeb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.780143 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.780127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c425ab8-0da3-4029-8406-1667e9262aeb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.785573 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.785511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24255\" (UniqueName: \"kubernetes.io/projected/6c425ab8-0da3-4029-8406-1667e9262aeb-kube-api-access-24255\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qglqf\" (UID: \"6c425ab8-0da3-4029-8406-1667e9262aeb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.849636 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.849600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:40.992153 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:40.992122 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf"] Jul 23 11:35:40.993813 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:35:40.993755 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c425ab8_0da3_4029_8406_1667e9262aeb.slice/crio-99f6682a066ad66bab89782319b9b5dece077d35915a02ecc26f3f22f4918faf WatchSource:0}: Error finding container 99f6682a066ad66bab89782319b9b5dece077d35915a02ecc26f3f22f4918faf: Status 404 returned error can't find the container with id 99f6682a066ad66bab89782319b9b5dece077d35915a02ecc26f3f22f4918faf Jul 23 11:35:41.501209 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:41.501150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" event={"ID":"6c425ab8-0da3-4029-8406-1667e9262aeb","Type":"ContainerStarted","Data":"99f6682a066ad66bab89782319b9b5dece077d35915a02ecc26f3f22f4918faf"} Jul 23 11:35:42.508722 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:42.508689 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" event={"ID":"6c425ab8-0da3-4029-8406-1667e9262aeb","Type":"ContainerStarted","Data":"1d5a3ba0f33d625180af56bde4df18902ac6fe39a314b6a59979d485bd07b79b"} Jul 23 11:35:42.528218 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:42.528167 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" podStartSLOduration=1.401272283 podStartE2EDuration="2.528151593s" totalImagesPullingTime="1.12687931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:42.526848554 +0000 UTC m=+536.508846057" watchObservedRunningTime="2026-07-23 11:35:42.528151593 +0000 UTC m=+536.510149135" Jul 23 11:35:47.357542 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:47.357500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:50.849927 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:50.849888 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:50.850407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:50.849954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:50.862372 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:50.862344 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:51.567395 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:51.567367 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qglqf" Jul 23 11:35:53.890209 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:53.890152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:35:59.676563 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:35:59.676519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:06.325844 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:06.325806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:18.268853 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:18.268817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:20.580716 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:20.580676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:20.681951 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:20.681913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:46.444577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:46.444547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:36:46.447122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:36:46.447102 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:37:33.897426 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:33.897388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:37:33.946991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:33.946959 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d8df7d8c5-dtj2h"] Jul 23 11:37:33.950633 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:33.950616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:33.958506 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:33.958482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d8df7d8c5-dtj2h"] Jul 23 11:37:34.096439 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.096405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvh4f\" (UniqueName: \"kubernetes.io/projected/31ae84b2-8fdb-40c2-b739-e2651c7e1815-kube-api-access-pvh4f\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.096583 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.096504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31ae84b2-8fdb-40c2-b739-e2651c7e1815-tls-cert\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.197959 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.197885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvh4f\" (UniqueName: \"kubernetes.io/projected/31ae84b2-8fdb-40c2-b739-e2651c7e1815-kube-api-access-pvh4f\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.198083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.197957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31ae84b2-8fdb-40c2-b739-e2651c7e1815-tls-cert\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.200373 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.200345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31ae84b2-8fdb-40c2-b739-e2651c7e1815-tls-cert\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.206058 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.206033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvh4f\" (UniqueName: \"kubernetes.io/projected/31ae84b2-8fdb-40c2-b739-e2651c7e1815-kube-api-access-pvh4f\") pod \"authorino-7d8df7d8c5-dtj2h\" (UID: \"31ae84b2-8fdb-40c2-b739-e2651c7e1815\") " pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.260944 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.260915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" Jul 23 11:37:34.409053 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.409032 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d8df7d8c5-dtj2h"] Jul 23 11:37:34.410722 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:37:34.410695 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ae84b2_8fdb_40c2_b739_e2651c7e1815.slice/crio-bd8f39cc7e3573545f519e2c3d4ea9296bca3273930d5b6f391ee599e644a716 WatchSource:0}: Error finding container bd8f39cc7e3573545f519e2c3d4ea9296bca3273930d5b6f391ee599e644a716: Status 404 returned error can't find the container with id bd8f39cc7e3573545f519e2c3d4ea9296bca3273930d5b6f391ee599e644a716 Jul 23 11:37:34.411931 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:34.411914 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:37:35.155598 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.155504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" event={"ID":"31ae84b2-8fdb-40c2-b739-e2651c7e1815","Type":"ContainerStarted","Data":"b4c082b1389d244df10b755ecffe690c0d27bf1a390bbf5eb174ee81a5ed991c"} Jul 23 11:37:35.155598 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.155560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" event={"ID":"31ae84b2-8fdb-40c2-b739-e2651c7e1815","Type":"ContainerStarted","Data":"bd8f39cc7e3573545f519e2c3d4ea9296bca3273930d5b6f391ee599e644a716"} Jul 23 11:37:35.174886 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.174811 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d8df7d8c5-dtj2h" podStartSLOduration=1.705376199 podStartE2EDuration="2.174789251s" totalImagesPullingTime="469.413052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:35.172493545 +0000 UTC m=+649.154491049" watchObservedRunningTime="2026-07-23 11:37:35.174789251 +0000 UTC m=+649.156786757" Jul 23 11:37:35.207562 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.207523 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:37:35.208120 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.208073 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" podUID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" containerName="authorino" containerID="cri-o://7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a" gracePeriod=30 Jul 23 11:37:35.398038 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.397947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:37:35.474350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.474319 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:37:35.533912 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.533881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:37:35.611626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.611592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgn7h\" (UniqueName: \"kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h\") pod \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " Jul 23 11:37:35.611805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.611657 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert\") pod \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\" (UID: \"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631\") " Jul 23 11:37:35.613954 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.613911 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h" (OuterVolumeSpecName: "kube-api-access-zgn7h") pod "69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" (UID: "69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631"). InnerVolumeSpecName "kube-api-access-zgn7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:35.628719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.628688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" (UID: "69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:35.632829 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.632804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:37:35.713141 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.713064 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-tls-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:37:35.713141 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:35.713093 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgn7h\" (UniqueName: \"kubernetes.io/projected/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631-kube-api-access-zgn7h\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:37:36.163545 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.163504 2564 generic.go:356] "Generic (PLEG): container finished" podID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" containerID="7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a" exitCode=0 Jul 23 11:37:36.164016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.163553 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" Jul 23 11:37:36.164016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.163552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" event={"ID":"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631","Type":"ContainerDied","Data":"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a"} Jul 23 11:37:36.164016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.163594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc8f5f54-cl2pz" event={"ID":"69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631","Type":"ContainerDied","Data":"f32064ccc9728f8b91c24696d65e0562f88cbc17349dbbe9b30cd50c581c7dee"} Jul 23 11:37:36.164016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.163612 2564 scope.go:117] "RemoveContainer" containerID="7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a" Jul 23 11:37:36.175527 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.175509 2564 scope.go:117] "RemoveContainer" containerID="7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a" Jul 23 11:37:36.175877 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:37:36.175847 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a\": container with ID starting with 7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a not found: ID does not exist" containerID="7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a" Jul 23 11:37:36.176002 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.175886 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a"} err="failed to get container status \"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a\": rpc error: code = NotFound desc = could not find container \"7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a\": container with ID starting with 7b67ac80052c1bb11498b8b189e9d2e33c2a3be01bdf40636ce30185280f2c4a not found: ID does not exist" Jul 23 11:37:36.194154 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.194127 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:37:36.204399 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.204367 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cbc8f5f54-cl2pz"] Jul 23 11:37:36.543463 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:37:36.543428 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" path="/var/lib/kubelet/pods/69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631/volumes" Jul 23 11:38:05.125637 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:38:05.125576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:38:05.226027 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:38:05.225991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:39:22.615681 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.615644 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:22.616189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.616072 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" containerName="authorino" Jul 23 11:39:22.616189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.616083 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" containerName="authorino" Jul 23 11:39:22.616189 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.616146 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c8f731-8ab5-4ab0-b6e2-5b2b2fe50631" containerName="authorino" Jul 23 11:39:22.619020 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.619004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.621490 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.621470 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 11:39:22.622347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.622334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 11:39:22.622396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.622381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bxwkd\"" Jul 23 11:39:22.629795 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.629745 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:22.712167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.712123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jvd\" (UniqueName: \"kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.712360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.712244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.813334 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.813293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jvd\" (UniqueName: \"kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.813523 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.813350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.815912 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.815892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.823038 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.823001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jvd\" (UniqueName: \"kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd\") pod \"maas-controller-79c669ccc6-7tmn6\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:22.930324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:22.930227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:23.078090 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:23.078062 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:23.080304 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:39:23.080277 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde855660_eb4e_4ccd_ab8f_f6b64c35ce9d.slice/crio-683830292e0347e23a1c45d1ad8cc435b34d3834b6afc9b3f5de4940c8f13cce WatchSource:0}: Error finding container 683830292e0347e23a1c45d1ad8cc435b34d3834b6afc9b3f5de4940c8f13cce: Status 404 returned error can't find the container with id 683830292e0347e23a1c45d1ad8cc435b34d3834b6afc9b3f5de4940c8f13cce Jul 23 11:39:23.798405 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:23.798368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" event={"ID":"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d","Type":"ContainerStarted","Data":"683830292e0347e23a1c45d1ad8cc435b34d3834b6afc9b3f5de4940c8f13cce"} Jul 23 11:39:25.812089 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:25.812049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" event={"ID":"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d","Type":"ContainerStarted","Data":"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602"} Jul 23 11:39:25.812529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:25.812144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:25.832369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:25.832317 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" podStartSLOduration=1.184907503 podStartE2EDuration="3.832298077s" totalImagesPullingTime="2.647390574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:39:25.828954998 +0000 UTC m=+759.810952501" watchObservedRunningTime="2026-07-23 11:39:25.832298077 +0000 UTC m=+759.814295584" Jul 23 11:39:36.820477 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:36.820447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:41.382462 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.382427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:39:41.446773 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.446732 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:41.446968 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.446946 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" podUID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" containerName="manager" containerID="cri-o://305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602" gracePeriod=30 Jul 23 11:39:41.697118 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.697092 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:41.794052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.794025 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert\") pod \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " Jul 23 11:39:41.794201 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.794098 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7jvd\" (UniqueName: \"kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd\") pod \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\" (UID: \"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d\") " Jul 23 11:39:41.796285 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.796259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd" (OuterVolumeSpecName: "kube-api-access-j7jvd") pod "de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" (UID: "de855660-eb4e-4ccd-ab8f-f6b64c35ce9d"). InnerVolumeSpecName "kube-api-access-j7jvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:39:41.796403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.796323 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" (UID: "de855660-eb4e-4ccd-ab8f-f6b64c35ce9d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:39:41.895577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.895550 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7jvd\" (UniqueName: \"kubernetes.io/projected/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-kube-api-access-j7jvd\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:39:41.895577 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.895574 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d-webhook-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:39:41.904940 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.904911 2564 generic.go:356] "Generic (PLEG): container finished" podID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" containerID="305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602" exitCode=0 Jul 23 11:39:41.905060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.904959 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" Jul 23 11:39:41.905060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.904967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" event={"ID":"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d","Type":"ContainerDied","Data":"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602"} Jul 23 11:39:41.905060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.905027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-7tmn6" event={"ID":"de855660-eb4e-4ccd-ab8f-f6b64c35ce9d","Type":"ContainerDied","Data":"683830292e0347e23a1c45d1ad8cc435b34d3834b6afc9b3f5de4940c8f13cce"} Jul 23 11:39:41.905060 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.905052 2564 scope.go:117] "RemoveContainer" containerID="305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602" Jul 23 11:39:41.915269 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.915252 2564 scope.go:117] "RemoveContainer" containerID="305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602" Jul 23 11:39:41.915543 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:39:41.915523 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602\": container with ID starting with 305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602 not found: ID does not exist" containerID="305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602" Jul 23 11:39:41.915602 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.915552 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602"} err="failed to get container status \"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602\": rpc error: code = NotFound desc = could not find container \"305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602\": container with ID starting with 305b41955f62564774af0d8f3453478ec510b9babb10b9a0f71381c0a9666602 not found: ID does not exist" Jul 23 11:39:41.925222 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.925198 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:41.928359 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:41.928315 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-7tmn6"] Jul 23 11:39:42.544915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.544878 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" path="/var/lib/kubelet/pods/de855660-eb4e-4ccd-ab8f-f6b64c35ce9d/volumes" Jul 23 11:39:42.643894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.643867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:39:42.708664 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.708629 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79c669ccc6-md9jl"] Jul 23 11:39:42.709041 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.709028 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" containerName="manager" Jul 23 11:39:42.709108 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.709042 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" containerName="manager" Jul 23 11:39:42.709148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.709124 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de855660-eb4e-4ccd-ab8f-f6b64c35ce9d" containerName="manager" Jul 23 11:39:42.713591 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.713571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.716179 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.716114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bxwkd\"" Jul 23 11:39:42.717975 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.717584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 11:39:42.717975 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.717585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 11:39:42.720489 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.720472 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-md9jl"] Jul 23 11:39:42.803008 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.802939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77a06d8-fd77-4f51-b333-abdd663c60d4-webhook-cert\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.803135 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.803061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgcp\" (UniqueName: \"kubernetes.io/projected/a77a06d8-fd77-4f51-b333-abdd663c60d4-kube-api-access-xjgcp\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.904245 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.904216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77a06d8-fd77-4f51-b333-abdd663c60d4-webhook-cert\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.904397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.904300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgcp\" (UniqueName: \"kubernetes.io/projected/a77a06d8-fd77-4f51-b333-abdd663c60d4-kube-api-access-xjgcp\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.906631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.906608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77a06d8-fd77-4f51-b333-abdd663c60d4-webhook-cert\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:42.913165 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:42.913138 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgcp\" (UniqueName: \"kubernetes.io/projected/a77a06d8-fd77-4f51-b333-abdd663c60d4-kube-api-access-xjgcp\") pod \"maas-controller-79c669ccc6-md9jl\" (UID: \"a77a06d8-fd77-4f51-b333-abdd663c60d4\") " pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:43.026033 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.026001 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:43.369064 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.367400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79c669ccc6-md9jl"] Jul 23 11:39:43.370837 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:39:43.370811 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77a06d8_fd77_4f51_b333_abdd663c60d4.slice/crio-03b8483039be9bd074dadc6785142044b2ab0bca1106ed4ce9885321ace5fe01 WatchSource:0}: Error finding container 03b8483039be9bd074dadc6785142044b2ab0bca1106ed4ce9885321ace5fe01: Status 404 returned error can't find the container with id 03b8483039be9bd074dadc6785142044b2ab0bca1106ed4ce9885321ace5fe01 Jul 23 11:39:43.919922 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.919889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-md9jl" event={"ID":"a77a06d8-fd77-4f51-b333-abdd663c60d4","Type":"ContainerStarted","Data":"27a1ef7e9a3f545502116dae27b472f9bc1505f03e85334ddf914bc1fe7795c7"} Jul 23 11:39:43.920352 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.919945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79c669ccc6-md9jl" event={"ID":"a77a06d8-fd77-4f51-b333-abdd663c60d4","Type":"ContainerStarted","Data":"03b8483039be9bd074dadc6785142044b2ab0bca1106ed4ce9885321ace5fe01"} Jul 23 11:39:43.920352 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.920008 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:39:43.937158 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:43.937109 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79c669ccc6-md9jl" podStartSLOduration=1.5328551959999999 podStartE2EDuration="1.937093713s" totalImagesPullingTime="404.238517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:39:43.935444171 +0000 UTC m=+777.917441678" watchObservedRunningTime="2026-07-23 11:39:43.937093713 +0000 UTC m=+777.919091216" Jul 23 11:39:54.927184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:39:54.927155 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79c669ccc6-md9jl" Jul 23 11:40:04.069195 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:04.069161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:18.105619 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:18.105538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:18.208510 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:18.208476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:18.330970 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:18.330930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:23.979986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:23.979949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:24.085114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:24.085076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:24.179049 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:24.179017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:24.332170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:24.332091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:30.158405 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:30.158375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:40:30.262281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:40:30.262252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:41:46.483098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:41:46.483070 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:41:46.486656 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:41:46.486638 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:43:31.152518 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:43:31.152435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:44:22.713755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:22.713721 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:44:22.714270 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:22.713975 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" podUID="97f78146-0176-4535-820e-cbef8a278975" containerName="manager" containerID="cri-o://22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af" gracePeriod=10 Jul 23 11:44:24.766155 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.766125 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:44:24.898446 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.898352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbddp\" (UniqueName: \"kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp\") pod \"97f78146-0176-4535-820e-cbef8a278975\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " Jul 23 11:44:24.898613 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.898481 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume\") pod \"97f78146-0176-4535-820e-cbef8a278975\" (UID: \"97f78146-0176-4535-820e-cbef8a278975\") " Jul 23 11:44:24.898931 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.898904 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97f78146-0176-4535-820e-cbef8a278975" (UID: "97f78146-0176-4535-820e-cbef8a278975"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:44:24.900475 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.900437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp" (OuterVolumeSpecName: "kube-api-access-jbddp") pod "97f78146-0176-4535-820e-cbef8a278975" (UID: "97f78146-0176-4535-820e-cbef8a278975"). InnerVolumeSpecName "kube-api-access-jbddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:44:24.999831 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.999800 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f78146-0176-4535-820e-cbef8a278975-extensions-socket-volume\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:44:24.999831 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:24.999824 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbddp\" (UniqueName: \"kubernetes.io/projected/97f78146-0176-4535-820e-cbef8a278975-kube-api-access-jbddp\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:44:25.520699 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.520661 2564 generic.go:356] "Generic (PLEG): container finished" podID="97f78146-0176-4535-820e-cbef8a278975" containerID="22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af" exitCode=0 Jul 23 11:44:25.520910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.520726 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" Jul 23 11:44:25.520910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.520743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" event={"ID":"97f78146-0176-4535-820e-cbef8a278975","Type":"ContainerDied","Data":"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af"} Jul 23 11:44:25.520910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.520844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6" event={"ID":"97f78146-0176-4535-820e-cbef8a278975","Type":"ContainerDied","Data":"f398dce414a273cd02e1b3d9e1fa058a9d1e2c2add557f1a7236a169264250c0"} Jul 23 11:44:25.520910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.520870 2564 scope.go:117] "RemoveContainer" containerID="22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af" Jul 23 11:44:25.531588 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.531569 2564 scope.go:117] "RemoveContainer" containerID="22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af" Jul 23 11:44:25.531855 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:44:25.531835 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af\": container with ID starting with 22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af not found: ID does not exist" containerID="22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af" Jul 23 11:44:25.531924 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.531876 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af"} err="failed to get container status \"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af\": rpc error: code = NotFound desc = could not find container \"22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af\": container with ID starting with 22ecab999b015aeaf2cfcb05183dee4fdf0d88850e6ba97bfd9406b74d6c14af not found: ID does not exist" Jul 23 11:44:25.544016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.543996 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:44:25.547404 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:25.547380 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrsn6"] Jul 23 11:44:26.543023 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:44:26.542989 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f78146-0176-4535-820e-cbef8a278975" path="/var/lib/kubelet/pods/97f78146-0176-4535-820e-cbef8a278975/volumes" Jul 23 11:45:00.146130 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.146091 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p"] Jul 23 11:45:00.146626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.146511 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f78146-0176-4535-820e-cbef8a278975" containerName="manager" Jul 23 11:45:00.146626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.146521 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f78146-0176-4535-820e-cbef8a278975" containerName="manager" Jul 23 11:45:00.146626 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.146600 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f78146-0176-4535-820e-cbef8a278975" containerName="manager" Jul 23 11:45:00.150921 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.150904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" Jul 23 11:45:00.154116 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.154083 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:45:00.156739 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.156715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p"] Jul 23 11:45:00.161894 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.161878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" Jul 23 11:45:00.299362 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.299339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p"] Jul 23 11:45:00.301977 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:45:00.301953 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1677f2bc_bf2d_44ed_a536_05ef54238942.slice/crio-397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03 WatchSource:0}: Error finding container 397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03: Status 404 returned error can't find the container with id 397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03 Jul 23 11:45:00.304184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.304169 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:45:00.715476 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:00.715440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" event={"ID":"1677f2bc-bf2d-44ed-a536-05ef54238942","Type":"ContainerStarted","Data":"397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03"} Jul 23 11:45:05.750969 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:05.750940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" event={"ID":"1677f2bc-bf2d-44ed-a536-05ef54238942","Type":"ContainerStarted","Data":"eb40f7bbc96a010054f4f698409fcb8305246030cd8f8e1cb0f247c146d8c2b8"} Jul 23 11:45:05.767645 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:05.767600 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" podStartSLOduration=0.843005179 podStartE2EDuration="5.767587776s" totalImagesPullingTime="4.924582597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:05.765260529 +0000 UTC m=+1099.747258033" watchObservedRunningTime="2026-07-23 11:45:05.767587776 +0000 UTC m=+1099.749585279" Jul 23 11:45:06.758144 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:06.758105 2564 generic.go:356] "Generic (PLEG): container finished" podID="1677f2bc-bf2d-44ed-a536-05ef54238942" containerID="eb40f7bbc96a010054f4f698409fcb8305246030cd8f8e1cb0f247c146d8c2b8" exitCode=0 Jul 23 11:45:06.758542 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:06.758194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" event={"ID":"1677f2bc-bf2d-44ed-a536-05ef54238942","Type":"ContainerDied","Data":"eb40f7bbc96a010054f4f698409fcb8305246030cd8f8e1cb0f247c146d8c2b8"} Jul 23 11:45:07.902061 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:07.902042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" Jul 23 11:45:08.770528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:08.770495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" event={"ID":"1677f2bc-bf2d-44ed-a536-05ef54238942","Type":"ContainerDied","Data":"397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03"} Jul 23 11:45:08.770671 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:08.770547 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="397089328a818325d1ee5cf43064f88e331e8303cd0160d4b1c61ba09aed4d03" Jul 23 11:45:08.770671 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:08.770508 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-cdh2p" Jul 23 11:45:23.909454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.909416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr"] Jul 23 11:45:23.910033 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.910015 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1677f2bc-bf2d-44ed-a536-05ef54238942" containerName="cleanup" Jul 23 11:45:23.910119 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.910036 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1677f2bc-bf2d-44ed-a536-05ef54238942" containerName="cleanup" Jul 23 11:45:23.910180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.910145 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1677f2bc-bf2d-44ed-a536-05ef54238942" containerName="cleanup" Jul 23 11:45:23.916182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.916163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:23.919057 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.919036 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jhmwr\"" Jul 23 11:45:23.927678 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:23.927655 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr"] Jul 23 11:45:24.005946 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.005903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.006101 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.005973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l92dh\" (UniqueName: \"kubernetes.io/projected/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-kube-api-access-l92dh\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.107312 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.107277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.107481 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.107332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l92dh\" (UniqueName: \"kubernetes.io/projected/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-kube-api-access-l92dh\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.107786 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.107741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.116710 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.116681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l92dh\" (UniqueName: \"kubernetes.io/projected/7c4abd77-94f1-479a-8a73-36e1b5d3f59a-kube-api-access-l92dh\") pod \"kuadrant-operator-controller-manager-86878c769f-642dr\" (UID: \"7c4abd77-94f1-479a-8a73-36e1b5d3f59a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.226754 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.226728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.374946 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.374922 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr"] Jul 23 11:45:24.376744 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:45:24.376722 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4abd77_94f1_479a_8a73_36e1b5d3f59a.slice/crio-30a4c55039ed00fe360b32a7eac0afd78335a94e0d0ee9ac8c03c145d202a9f0 WatchSource:0}: Error finding container 30a4c55039ed00fe360b32a7eac0afd78335a94e0d0ee9ac8c03c145d202a9f0: Status 404 returned error can't find the container with id 30a4c55039ed00fe360b32a7eac0afd78335a94e0d0ee9ac8c03c145d202a9f0 Jul 23 11:45:24.866927 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.866888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" event={"ID":"7c4abd77-94f1-479a-8a73-36e1b5d3f59a","Type":"ContainerStarted","Data":"bbdc3c733ca5094b63028892fd7a47e3cd657c5175a2817a72254d2ed915b422"} Jul 23 11:45:24.867275 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.866974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" event={"ID":"7c4abd77-94f1-479a-8a73-36e1b5d3f59a","Type":"ContainerStarted","Data":"30a4c55039ed00fe360b32a7eac0afd78335a94e0d0ee9ac8c03c145d202a9f0"} Jul 23 11:45:24.867275 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.867135 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:45:24.887747 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:24.887695 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" podStartSLOduration=1.887679704 podStartE2EDuration="1.887679704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:24.886040086 +0000 UTC m=+1118.868037591" watchObservedRunningTime="2026-07-23 11:45:24.887679704 +0000 UTC m=+1118.869677206" Jul 23 11:45:35.875308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:45:35.875279 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-642dr" Jul 23 11:46:15.239434 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:46:15.239354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:46:46.520742 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:46:46.520713 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:46:46.525000 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:46:46.524981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:47:19.453791 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:47:19.453725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:48:05.181657 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:48:05.181605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:48:10.859202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:48:10.859163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:48:36.980374 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:48:36.980322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:48:38.495258 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:48:38.495222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:49:04.426260 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:49:04.426220 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:49:04.480152 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:49:04.480115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:51:46.560432 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:51:46.560401 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:51:46.565908 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:51:46.565886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:51:53.709090 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:51:53.709051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:51:53.796992 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:51:53.796965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:16.865116 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:16.865033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:20.022216 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:20.022180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:20.117552 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:20.117523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:20.468997 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:20.468907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:26.191653 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:26.191614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:26.291383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:26.291349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:26.409618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:26.409583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:26.733701 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:26.733654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:27.044299 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:27.044217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:29.132866 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:29.132826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:29.229318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:29.229271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:29.371347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:29.371316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:29.676184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:29.676154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:29.983211 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:29.983171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:30.080155 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:30.080117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:52:54.735987 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.735957 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl"] Jul 23 11:52:54.739303 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.739287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:54.741675 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.741649 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:52:54.745793 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.745753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl"] Jul 23 11:52:54.882191 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.882155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl\" (UID: \"061a1efa-52c7-4f84-b56e-50703dba48a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:54.983646 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.983607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl\" (UID: \"061a1efa-52c7-4f84-b56e-50703dba48a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:54.984333 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:54.984316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl\" (UID: \"061a1efa-52c7-4f84-b56e-50703dba48a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:55.051350 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:55.051265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:55.190292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:55.190268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl"] Jul 23 11:52:55.192702 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:52:55.192671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061a1efa_52c7_4f84_b56e_50703dba48a3.slice/crio-a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b WatchSource:0}: Error finding container a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b: Status 404 returned error can't find the container with id a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b Jul 23 11:52:55.541396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:55.541367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" event={"ID":"061a1efa-52c7-4f84-b56e-50703dba48a3","Type":"ContainerStarted","Data":"3bc8bb0ef197857e3f0e013ea14fdb156083a7fab24eaa3cb32cbc0fd61d4b44"} Jul 23 11:52:55.541547 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:55.541416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" event={"ID":"061a1efa-52c7-4f84-b56e-50703dba48a3","Type":"ContainerStarted","Data":"a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b"} Jul 23 11:52:55.556501 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:55.556450 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" podStartSLOduration=1.556434257 podStartE2EDuration="1.556434257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:52:55.556222121 +0000 UTC m=+1569.538219637" watchObservedRunningTime="2026-07-23 11:52:55.556434257 +0000 UTC m=+1569.538431760" Jul 23 11:52:56.548025 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:56.547993 2564 generic.go:356] "Generic (PLEG): container finished" podID="061a1efa-52c7-4f84-b56e-50703dba48a3" containerID="3bc8bb0ef197857e3f0e013ea14fdb156083a7fab24eaa3cb32cbc0fd61d4b44" exitCode=0 Jul 23 11:52:56.548407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:56.548070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" event={"ID":"061a1efa-52c7-4f84-b56e-50703dba48a3","Type":"ContainerDied","Data":"3bc8bb0ef197857e3f0e013ea14fdb156083a7fab24eaa3cb32cbc0fd61d4b44"} Jul 23 11:52:57.682661 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:57.682637 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:57.810010 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:57.809926 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca\") pod \"061a1efa-52c7-4f84-b56e-50703dba48a3\" (UID: \"061a1efa-52c7-4f84-b56e-50703dba48a3\") " Jul 23 11:52:57.810260 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:57.810238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "061a1efa-52c7-4f84-b56e-50703dba48a3" (UID: "061a1efa-52c7-4f84-b56e-50703dba48a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:52:57.911871 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:57.911834 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/061a1efa-52c7-4f84-b56e-50703dba48a3-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:52:58.561449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:58.561416 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" Jul 23 11:52:58.561625 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:58.561457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl" event={"ID":"061a1efa-52c7-4f84-b56e-50703dba48a3","Type":"ContainerDied","Data":"a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b"} Jul 23 11:52:58.561625 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:52:58.561510 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b8d6200907b0addca89516532cfeb65b8fa42ce67c8eaf094b1a4aca68504b" Jul 23 11:53:05.276412 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:05.276380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:09.816003 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:09.815968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:09.827098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:09.827069 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl"] Jul 23 11:53:09.829340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:09.829318 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4ab95e-rg6pl"] Jul 23 11:53:09.979719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:09.979678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:10.221477 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:10.221449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:10.500180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:10.500085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:10.543161 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:10.543126 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061a1efa-52c7-4f84-b56e-50703dba48a3" path="/var/lib/kubelet/pods/061a1efa-52c7-4f84-b56e-50703dba48a3/volumes" Jul 23 11:53:10.601915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:10.601885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:10.702372 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:10.702340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:11.012016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.011982 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:53:11.012446 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.012432 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061a1efa-52c7-4f84-b56e-50703dba48a3" containerName="revoke-keys" Jul 23 11:53:11.012493 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.012448 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="061a1efa-52c7-4f84-b56e-50703dba48a3" containerName="revoke-keys" Jul 23 11:53:11.012528 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.012523 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="061a1efa-52c7-4f84-b56e-50703dba48a3" containerName="revoke-keys" Jul 23 11:53:11.015750 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.015732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.018166 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.018142 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:53:11.018298 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.018237 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d6a2356d-gw-openshift-default-dockercfg-wnfnm\"" Jul 23 11:53:11.028869 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.028842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:53:11.032681 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.032655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:11.129543 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpx6\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.129913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.129820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpx6\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.230686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.231074 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.231074 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.230965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.231172 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.231100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.231172 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.231153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.231454 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.231436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.233112 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.233085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.233256 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.233241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.239034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.239010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.239135 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.239060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpx6\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6\") pod \"e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.327240 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.327149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:11.372805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.372747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:11.476359 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.476327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:53:11.476919 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:53:11.476889 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b489c3_2575_4ac9_ac67_ec7471187eeb.slice/crio-445e81dd6653b63b72ebe9c7e5105856b3a96feb2c99eacd10b8eb1af6d4706f WatchSource:0}: Error finding container 445e81dd6653b63b72ebe9c7e5105856b3a96feb2c99eacd10b8eb1af6d4706f: Status 404 returned error can't find the container with id 445e81dd6653b63b72ebe9c7e5105856b3a96feb2c99eacd10b8eb1af6d4706f Jul 23 11:53:11.479281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.479262 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:53:11.638485 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:11.638396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerStarted","Data":"445e81dd6653b63b72ebe9c7e5105856b3a96feb2c99eacd10b8eb1af6d4706f"} Jul 23 11:53:14.425980 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.425939 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:14.426347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.426034 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:14.426347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.426083 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:14.657403 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.657371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerStarted","Data":"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5"} Jul 23 11:53:14.663142 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.663115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:14.678583 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:14.678494 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podStartSLOduration=1.732297811 podStartE2EDuration="4.678473405s" totalImagesPullingTime="2.946175594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:53:14.675842638 +0000 UTC m=+1588.657840145" watchObservedRunningTime="2026-07-23 11:53:14.678473405 +0000 UTC m=+1588.660470907" Jul 23 11:53:15.327755 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:15.327716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:15.332991 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:15.332963 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:15.663387 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:15.663296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:15.664436 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:15.664418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:16.140367 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:16.140334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:16.258976 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:16.258949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:16.360349 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:16.360316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:16.872413 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:16.872374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:16.973713 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:16.973676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:28.744287 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.744255 2564 generic.go:356] "Generic (PLEG): container finished" podID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerID="cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5" exitCode=0 Jul 23 11:53:28.744701 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.744325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerDied","Data":"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5"} Jul 23 11:53:28.744701 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.744664 2564 scope.go:117] "RemoveContainer" containerID="cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5" Jul 23 11:53:28.745609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.745581 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:28.745677 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.745650 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:28.745714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:28.745679 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:29.753096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:29.753062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerStarted","Data":"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa"} Jul 23 11:53:30.328301 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:30.328268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:30.329447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:30.329419 2564 patch_prober.go:28] interesting pod/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 11:53:30.329572 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:30.329468 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 11:53:30.663613 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:30.663502 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:31.328594 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:31.328549 2564 patch_prober.go:28] interesting pod/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 11:53:31.329069 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:31.328619 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 11:53:32.328119 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:32.328082 2564 patch_prober.go:28] interesting pod/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 11:53:32.328308 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:32.328137 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 11:53:33.328088 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:33.328045 2564 patch_prober.go:28] interesting pod/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 11:53:33.328507 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:33.328110 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 11:53:34.331525 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:34.331496 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:34.332596 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:34.332577 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:53:41.198707 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.198659 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z"] Jul 23 11:53:41.203942 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.203923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:41.208071 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.208040 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:53:41.209520 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.209496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z"] Jul 23 11:53:41.309063 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.309030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z\" (UID: \"25501044-51c2-4fb7-ad5b-e018638ed8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:41.410092 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.410059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z\" (UID: \"25501044-51c2-4fb7-ad5b-e018638ed8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:41.410718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.410696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z\" (UID: \"25501044-51c2-4fb7-ad5b-e018638ed8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:41.516425 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.516392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:41.655150 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.655119 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z"] Jul 23 11:53:41.657123 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:53:41.657088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25501044_51c2_4fb7_ad5b_e018638ed8fa.slice/crio-30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47 WatchSource:0}: Error finding container 30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47: Status 404 returned error can't find the container with id 30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47 Jul 23 11:53:41.832314 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:41.832226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" event={"ID":"25501044-51c2-4fb7-ad5b-e018638ed8fa","Type":"ContainerStarted","Data":"30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47"} Jul 23 11:53:42.839956 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:42.839917 2564 generic.go:356] "Generic (PLEG): container finished" podID="25501044-51c2-4fb7-ad5b-e018638ed8fa" containerID="ccee51b4a384a1273a4f07777ee289e9ab1da0ad2634a5d3c1fe62ac892db330" exitCode=0 Jul 23 11:53:42.840377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:42.840008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" event={"ID":"25501044-51c2-4fb7-ad5b-e018638ed8fa","Type":"ContainerDied","Data":"ccee51b4a384a1273a4f07777ee289e9ab1da0ad2634a5d3c1fe62ac892db330"} Jul 23 11:53:43.988984 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:43.988959 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:44.134498 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.134387 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca\") pod \"25501044-51c2-4fb7-ad5b-e018638ed8fa\" (UID: \"25501044-51c2-4fb7-ad5b-e018638ed8fa\") " Jul 23 11:53:44.134754 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.134731 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25501044-51c2-4fb7-ad5b-e018638ed8fa" (UID: "25501044-51c2-4fb7-ad5b-e018638ed8fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:53:44.134855 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.134835 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25501044-51c2-4fb7-ad5b-e018638ed8fa-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:53:44.853313 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.853233 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" Jul 23 11:53:44.853448 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.853314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z" event={"ID":"25501044-51c2-4fb7-ad5b-e018638ed8fa","Type":"ContainerDied","Data":"30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47"} Jul 23 11:53:44.853448 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:44.853362 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e4b7be0dadb530e9179bc1e00854ba06fccc987b32bb3272d62d8045128a47" Jul 23 11:53:51.656324 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:51.656293 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:56.286634 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.286600 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z"] Jul 23 11:53:56.289674 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.289651 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a2356d-s859z"] Jul 23 11:53:56.314322 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.314294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:56.436168 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.436132 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:53:56.436472 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.436446 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" containerID="cri-o://5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa" gracePeriod=30 Jul 23 11:53:56.457272 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.457243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:56.544651 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.544569 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25501044-51c2-4fb7-ad5b-e018638ed8fa" path="/var/lib/kubelet/pods/25501044-51c2-4fb7-ad5b-e018638ed8fa/volumes" Jul 23 11:53:56.689451 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.689417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:56.931285 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:56.931199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:57.265849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.265821 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:53:57.266246 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.266234 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25501044-51c2-4fb7-ad5b-e018638ed8fa" containerName="revoke-keys" Jul 23 11:53:57.266292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.266248 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25501044-51c2-4fb7-ad5b-e018638ed8fa" containerName="revoke-keys" Jul 23 11:53:57.266369 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.266307 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="25501044-51c2-4fb7-ad5b-e018638ed8fa" containerName="revoke-keys" Jul 23 11:53:57.269732 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.269716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.272316 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.272292 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aee44b3e-openshift-default-dockercfg-chjrk\"" Jul 23 11:53:57.281962 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.281937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:53:57.293778 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.293740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:57.450905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.450870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.450919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.450943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fxb\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451358 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451358 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.451358 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.451238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552343 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fxb\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.552639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.553024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.553024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.553024 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.552994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.553166 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.553095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.553206 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.553166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.554810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.554792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.554896 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.554881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.560753 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.560731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.561124 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.561104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fxb\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb\") pod \"e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.582305 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.582285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:57.664724 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.664695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:57.732241 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.732218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:53:57.734416 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:53:57.734394 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37f8700_2b07_490c_90ef_4d92b288f507.slice/crio-30cf5f132d114e1966e9b177070d468940fef3794f785f25b216c41045f566db WatchSource:0}: Error finding container 30cf5f132d114e1966e9b177070d468940fef3794f785f25b216c41045f566db: Status 404 returned error can't find the container with id 30cf5f132d114e1966e9b177070d468940fef3794f785f25b216c41045f566db Jul 23 11:53:57.737102 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.737065 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:57.737187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.737130 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:57.737187 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.737160 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:53:57.931483 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.931383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerStarted","Data":"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14"} Jul 23 11:53:57.931483 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.931452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerStarted","Data":"30cf5f132d114e1966e9b177070d468940fef3794f785f25b216c41045f566db"} Jul 23 11:53:57.952529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:57.952478 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podStartSLOduration=0.952462109 podStartE2EDuration="952.462109ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:53:57.949614355 +0000 UTC m=+1631.931611858" watchObservedRunningTime="2026-07-23 11:53:57.952462109 +0000 UTC m=+1631.934459612" Jul 23 11:53:58.582720 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:58.582683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:58.587888 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:58.587857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:58.936897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:58.936801 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:58.937635 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:58.937613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:53:59.303194 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:59.303162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:59.402993 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:59.402958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:59.500052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:59.500021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:53:59.956817 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:53:59.956782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:00.057977 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:00.057941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:01.706085 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.706026 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:54:01.896662 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.896876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.896876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.896876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcpx6\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.896991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897184 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897136 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897291 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897186 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket\") pod \"17b489c3-2575-4ac9-ac67-ec7471187eeb\" (UID: \"17b489c3-2575-4ac9-ac67-ec7471187eeb\") " Jul 23 11:54:01.897291 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897278 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:54:01.897479 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897459 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:01.897550 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:01.897608 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897554 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b489c3-2575-4ac9-ac67-ec7471187eeb-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.897608 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897573 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.897714 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:01.897840 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.897821 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:01.899391 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.899361 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6" (OuterVolumeSpecName: "kube-api-access-lcpx6") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "kube-api-access-lcpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:54:01.899494 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.899386 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:54:01.899607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.899588 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:01.899607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.899594 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17b489c3-2575-4ac9-ac67-ec7471187eeb" (UID: "17b489c3-2575-4ac9-ac67-ec7471187eeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 11:54:01.956390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.956321 2564 generic.go:356] "Generic (PLEG): container finished" podID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerID="5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa" exitCode=0 Jul 23 11:54:01.956390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.956366 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" Jul 23 11:54:01.956390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.956383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerDied","Data":"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa"} Jul 23 11:54:01.956605 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.956438 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp" event={"ID":"17b489c3-2575-4ac9-ac67-ec7471187eeb","Type":"ContainerDied","Data":"445e81dd6653b63b72ebe9c7e5105856b3a96feb2c99eacd10b8eb1af6d4706f"} Jul 23 11:54:01.956605 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.956470 2564 scope.go:117] "RemoveContainer" containerID="5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa" Jul 23 11:54:01.966059 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.966044 2564 scope.go:117] "RemoveContainer" containerID="cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5" Jul 23 11:54:01.976515 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.976494 2564 scope.go:117] "RemoveContainer" containerID="5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa" Jul 23 11:54:01.976835 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:54:01.976813 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa\": container with ID starting with 5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa not found: ID does not exist" containerID="5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa" Jul 23 11:54:01.976910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.976844 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa"} err="failed to get container status \"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa\": rpc error: code = NotFound desc = could not find container \"5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa\": container with ID starting with 5de0c6ea1c7b45b1d4ad6cbcf08755d05d3068b1efd33e8476dd2a560ac1f1aa not found: ID does not exist" Jul 23 11:54:01.976910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.976862 2564 scope.go:117] "RemoveContainer" containerID="cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5" Jul 23 11:54:01.977123 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:54:01.977104 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5\": container with ID starting with cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5 not found: ID does not exist" containerID="cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5" Jul 23 11:54:01.977186 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.977132 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5"} err="failed to get container status \"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5\": rpc error: code = NotFound desc = could not find container \"cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5\": container with ID starting with cd40e5d93162d842c3e7bb62ac8efa07cc1ff54b43a8c2a65848745775aef7f5 not found: ID does not exist" Jul 23 11:54:01.979305 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.979286 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:54:01.981877 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.981860 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d6a2356d-gw-openshift-default-5466d6fb79-qspsp"] Jul 23 11:54:01.998689 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998670 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998692 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998705 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998717 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpx6\" (UniqueName: \"kubernetes.io/projected/17b489c3-2575-4ac9-ac67-ec7471187eeb-kube-api-access-lcpx6\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998730 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b489c3-2575-4ac9-ac67-ec7471187eeb-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998745 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:01.998828 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:01.998757 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b489c3-2575-4ac9-ac67-ec7471187eeb-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:02.542953 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:02.542919 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" path="/var/lib/kubelet/pods/17b489c3-2575-4ac9-ac67-ec7471187eeb/volumes" Jul 23 11:54:12.025986 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.025956 2564 generic.go:356] "Generic (PLEG): container finished" podID="f37f8700-2b07-490c-90ef-4d92b288f507" containerID="5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14" exitCode=0 Jul 23 11:54:12.026390 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.026029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerDied","Data":"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14"} Jul 23 11:54:12.026434 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.026402 2564 scope.go:117] "RemoveContainer" containerID="5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14" Jul 23 11:54:12.027377 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.027346 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:12.027536 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.027415 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:12.027536 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:12.027448 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:13.033622 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:13.033590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerStarted","Data":"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0"} Jul 23 11:54:13.584478 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:13.584444 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:13.584915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:13.584888 2564 patch_prober.go:28] interesting pod/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 23 11:54:13.585117 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:13.585087 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 23 11:54:13.936974 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:13.936884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:14.582915 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:14.582876 2564 patch_prober.go:28] interesting pod/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 23 11:54:14.583407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:14.582945 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 23 11:54:15.584083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:15.583616 2564 patch_prober.go:28] interesting pod/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 23 11:54:15.584083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:15.583689 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 23 11:54:16.583805 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:16.583740 2564 patch_prober.go:28] interesting pod/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 23 11:54:16.584013 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:16.583849 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 23 11:54:17.586642 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:17.586612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:17.587719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:17.587698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:25.209295 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209258 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r"] Jul 23 11:54:25.209809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209731 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:25.209809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209744 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:25.209809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209755 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:25.209809 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209775 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:25.209944 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.209840 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:25.212882 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.212866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:25.215442 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.215419 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:54:25.220262 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.220239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r"] Jul 23 11:54:25.309856 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.309825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r\" (UID: \"a99a0ef7-9c95-49ab-8763-56e6585d6b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:25.411266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.411224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r\" (UID: \"a99a0ef7-9c95-49ab-8763-56e6585d6b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:25.411884 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.411864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r\" (UID: \"a99a0ef7-9c95-49ab-8763-56e6585d6b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:25.525211 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.525178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:25.665317 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:25.665291 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r"] Jul 23 11:54:25.666828 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:54:25.666800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99a0ef7_9c95_49ab_8763_56e6585d6b64.slice/crio-c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523 WatchSource:0}: Error finding container c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523: Status 404 returned error can't find the container with id c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523 Jul 23 11:54:26.115820 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:26.115775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" event={"ID":"a99a0ef7-9c95-49ab-8763-56e6585d6b64","Type":"ContainerStarted","Data":"d1f68eb50727bd7f075c628aa6612372e99b1a3df0aa68b998321791d1aa6e2e"} Jul 23 11:54:26.116004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:26.115867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" event={"ID":"a99a0ef7-9c95-49ab-8763-56e6585d6b64","Type":"ContainerStarted","Data":"c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523"} Jul 23 11:54:26.133823 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:26.133757 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" podStartSLOduration=1.13374275 podStartE2EDuration="1.13374275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:54:26.131982484 +0000 UTC m=+1660.113979987" watchObservedRunningTime="2026-07-23 11:54:26.13374275 +0000 UTC m=+1660.115740268" Jul 23 11:54:27.123058 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:27.123023 2564 generic.go:356] "Generic (PLEG): container finished" podID="a99a0ef7-9c95-49ab-8763-56e6585d6b64" containerID="d1f68eb50727bd7f075c628aa6612372e99b1a3df0aa68b998321791d1aa6e2e" exitCode=0 Jul 23 11:54:27.123440 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:27.123117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" event={"ID":"a99a0ef7-9c95-49ab-8763-56e6585d6b64","Type":"ContainerDied","Data":"d1f68eb50727bd7f075c628aa6612372e99b1a3df0aa68b998321791d1aa6e2e"} Jul 23 11:54:28.267262 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:28.267240 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:28.336631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:28.336603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca\") pod \"a99a0ef7-9c95-49ab-8763-56e6585d6b64\" (UID: \"a99a0ef7-9c95-49ab-8763-56e6585d6b64\") " Jul 23 11:54:28.336974 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:28.336955 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a99a0ef7-9c95-49ab-8763-56e6585d6b64" (UID: "a99a0ef7-9c95-49ab-8763-56e6585d6b64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:54:28.438025 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:28.437963 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99a0ef7-9c95-49ab-8763-56e6585d6b64-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:29.137170 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:29.137134 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" event={"ID":"a99a0ef7-9c95-49ab-8763-56e6585d6b64","Type":"ContainerDied","Data":"c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523"} Jul 23 11:54:29.137340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:29.137182 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00c3976cb5d4875cdc4b77645e92274ee25638708d8e6fd32de908e798ca523" Jul 23 11:54:29.137340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:29.137151 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r" Jul 23 11:54:35.658248 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:35.658209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:40.290278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.290239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:40.305480 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.305453 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r"] Jul 23 11:54:40.307748 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.307730 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aee44b3e-dn89r"] Jul 23 11:54:40.542812 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.542719 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99a0ef7-9c95-49ab-8763-56e6585d6b64" path="/var/lib/kubelet/pods/a99a0ef7-9c95-49ab-8763-56e6585d6b64/volumes" Jul 23 11:54:40.557711 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.557686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:40.612973 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:40.612943 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:41.487550 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:41.487278 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:54:41.488040 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:41.487666 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" containerID="cri-o://e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0" gracePeriod=30 Jul 23 11:54:41.501321 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:41.501295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:41.740941 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:41.740849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:41.956235 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:41.956194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:42.259279 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.259251 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:54:42.259686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.259673 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99a0ef7-9c95-49ab-8763-56e6585d6b64" containerName="revoke-keys" Jul 23 11:54:42.259734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.259688 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99a0ef7-9c95-49ab-8763-56e6585d6b64" containerName="revoke-keys" Jul 23 11:54:42.259821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.259811 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99a0ef7-9c95-49ab-8763-56e6585d6b64" containerName="revoke-keys" Jul 23 11:54:42.259856 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.259824 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b489c3-2575-4ac9-ac67-ec7471187eeb" containerName="istio-proxy" Jul 23 11:54:42.263428 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.263406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.266610 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.266589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a4beeb76-openshift-default-dockercfg-trv9l\"" Jul 23 11:54:42.282231 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.282200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:54:42.294996 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.294969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:42.365160 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365160 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365396 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365572 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.365572 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.365458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhcs\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466410 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhcs\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.466673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.467589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.467589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.466956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.467589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.467166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.467589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.467220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.467589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.467236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.468987 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.468965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.469280 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.469262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.475338 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.475312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.475910 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.475887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhcs\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs\") pod \"e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.524843 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.524735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:42.573292 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.573257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:42.725906 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:54:42.725877 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a37ab87_26f1_4c7b_b305_19994c069c24.slice/crio-648d62ea66c1c5fef343134f005bed8e86aa08d2edc8ccd264636d303c198040 WatchSource:0}: Error finding container 648d62ea66c1c5fef343134f005bed8e86aa08d2edc8ccd264636d303c198040: Status 404 returned error can't find the container with id 648d62ea66c1c5fef343134f005bed8e86aa08d2edc8ccd264636d303c198040 Jul 23 11:54:42.726032 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.725990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:54:42.728157 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.728116 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:42.728253 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.728205 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:42.728290 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:42.728252 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:43.228114 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:43.228073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerStarted","Data":"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3"} Jul 23 11:54:43.228267 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:43.228140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerStarted","Data":"648d62ea66c1c5fef343134f005bed8e86aa08d2edc8ccd264636d303c198040"} Jul 23 11:54:43.255812 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:43.255726 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podStartSLOduration=1.255705747 podStartE2EDuration="1.255705747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:54:43.254476079 +0000 UTC m=+1677.236473575" watchObservedRunningTime="2026-07-23 11:54:43.255705747 +0000 UTC m=+1677.237703251" Jul 23 11:54:43.574032 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:43.573957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:43.579623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:43.579583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:44.233514 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:44.233481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:44.234347 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:44.234330 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:54:44.600098 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:44.600019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:44.691485 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:44.691459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:44.755987 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:44.755955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:45.361981 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:45.361952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:45.461816 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:45.461784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:54:46.749427 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.749405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:46.911733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911647 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911733 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911688 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7fxb\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911936 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.911979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.911962 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.912274 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912099 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo\") pod \"f37f8700-2b07-490c-90ef-4d92b288f507\" (UID: \"f37f8700-2b07-490c-90ef-4d92b288f507\") " Jul 23 11:54:46.912274 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912016 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:46.912274 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912156 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:46.912469 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912429 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:46.912469 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912448 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:46.912582 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912477 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:46.912582 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912518 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:54:46.912725 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.912696 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data" (OuterVolumeSpecName: "istio-data") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:46.914159 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.914138 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token" (OuterVolumeSpecName: "istio-token") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:54:46.914365 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.914346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 11:54:46.914670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.914650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:54:46.914794 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:46.914675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb" (OuterVolumeSpecName: "kube-api-access-r7fxb") pod "f37f8700-2b07-490c-90ef-4d92b288f507" (UID: "f37f8700-2b07-490c-90ef-4d92b288f507"). InnerVolumeSpecName "kube-api-access-r7fxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:54:47.013590 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013568 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013590 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013589 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013796 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013598 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fxb\" (UniqueName: \"kubernetes.io/projected/f37f8700-2b07-490c-90ef-4d92b288f507-kube-api-access-r7fxb\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013796 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013608 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37f8700-2b07-490c-90ef-4d92b288f507-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013796 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013617 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013796 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013624 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37f8700-2b07-490c-90ef-4d92b288f507-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.013796 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.013631 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37f8700-2b07-490c-90ef-4d92b288f507-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:54:47.252260 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.252230 2564 generic.go:356] "Generic (PLEG): container finished" podID="f37f8700-2b07-490c-90ef-4d92b288f507" containerID="e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0" exitCode=0 Jul 23 11:54:47.252449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.252278 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" Jul 23 11:54:47.252449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.252309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerDied","Data":"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0"} Jul 23 11:54:47.252449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.252348 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr" event={"ID":"f37f8700-2b07-490c-90ef-4d92b288f507","Type":"ContainerDied","Data":"30cf5f132d114e1966e9b177070d468940fef3794f785f25b216c41045f566db"} Jul 23 11:54:47.252449 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.252370 2564 scope.go:117] "RemoveContainer" containerID="e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0" Jul 23 11:54:47.262330 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.262308 2564 scope.go:117] "RemoveContainer" containerID="5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14" Jul 23 11:54:47.270360 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.270346 2564 scope.go:117] "RemoveContainer" containerID="e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0" Jul 23 11:54:47.270589 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:54:47.270571 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0\": container with ID starting with e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0 not found: ID does not exist" containerID="e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0" Jul 23 11:54:47.270629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.270596 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0"} err="failed to get container status \"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0\": rpc error: code = NotFound desc = could not find container \"e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0\": container with ID starting with e890bcf4762a97ef909bfcae66b7799b965b6f132e5fb581ce869b900a0df5f0 not found: ID does not exist" Jul 23 11:54:47.270629 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.270610 2564 scope.go:117] "RemoveContainer" containerID="5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14" Jul 23 11:54:47.270812 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:54:47.270793 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14\": container with ID starting with 5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14 not found: ID does not exist" containerID="5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14" Jul 23 11:54:47.270860 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.270817 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14"} err="failed to get container status \"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14\": rpc error: code = NotFound desc = could not find container \"5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14\": container with ID starting with 5ab69cfc2584ba5e0a77fc6b5195287c296fa4109992b8bbd9368b5e14518a14 not found: ID does not exist" Jul 23 11:54:47.274810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.274787 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:54:47.280419 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:47.280400 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aee44b3e-openshift-default-7697bdbd9c-dhvcr"] Jul 23 11:54:48.543545 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:48.543512 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" path="/var/lib/kubelet/pods/f37f8700-2b07-490c-90ef-4d92b288f507/volumes" Jul 23 11:54:59.234990 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.234957 2564 patch_prober.go:28] interesting pod/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 23 11:54:59.235388 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.235020 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 23 11:54:59.334278 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.334202 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerID="b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3" exitCode=0 Jul 23 11:54:59.334407 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.334288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerDied","Data":"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3"} Jul 23 11:54:59.334684 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.334670 2564 scope.go:117] "RemoveContainer" containerID="b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3" Jul 23 11:54:59.335718 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.335690 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:59.335808 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.335790 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:54:59.335868 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:54:59.335843 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:55:00.342213 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:00.342172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerStarted","Data":"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e"} Jul 23 11:55:00.573787 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:00.573736 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:00.574135 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:00.574094 2564 patch_prober.go:28] interesting pod/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 23 11:55:00.574340 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:00.574315 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 23 11:55:01.574895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:01.574466 2564 patch_prober.go:28] interesting pod/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 23 11:55:01.574895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:01.574527 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 23 11:55:02.574210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:02.574173 2564 patch_prober.go:28] interesting pod/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 23 11:55:02.574420 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:02.574232 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 23 11:55:03.575022 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:03.574970 2564 patch_prober.go:28] interesting pod/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 23 11:55:03.575509 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:03.575034 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 23 11:55:04.577744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:04.577716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:04.578206 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:04.577986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:04.578783 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:04.578752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:06.226281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:06.226249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:06.484700 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:06.484578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:07.048670 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:07.048640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:07.145553 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:07.145518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:09.542079 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542047 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92"] Jul 23 11:55:09.542470 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542456 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.542470 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542471 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.542543 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542478 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.542543 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542483 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.542609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542546 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.542609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.542557 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37f8700-2b07-490c-90ef-4d92b288f507" containerName="istio-proxy" Jul 23 11:55:09.544719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.544690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:09.547345 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.547322 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:55:09.558426 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.558406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92"] Jul 23 11:55:09.617965 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.617915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92\" (UID: \"f339c3c9-12a1-409e-81bd-cb4754b62d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:09.718911 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.718869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92\" (UID: \"f339c3c9-12a1-409e-81bd-cb4754b62d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:09.719491 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.719470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92\" (UID: \"f339c3c9-12a1-409e-81bd-cb4754b62d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:09.854689 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.854603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:09.998743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:09.998717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92"] Jul 23 11:55:10.000544 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:55:10.000518 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf339c3c9_12a1_409e_81bd_cb4754b62d0b.slice/crio-69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d WatchSource:0}: Error finding container 69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d: Status 404 returned error can't find the container with id 69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d Jul 23 11:55:10.405743 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:10.405709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" event={"ID":"f339c3c9-12a1-409e-81bd-cb4754b62d0b","Type":"ContainerStarted","Data":"24a8bf7e497d2ac36ba4bfe0a9a4e27fd5abef171a34771d0d98490291c73d48"} Jul 23 11:55:10.405938 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:10.405780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" event={"ID":"f339c3c9-12a1-409e-81bd-cb4754b62d0b","Type":"ContainerStarted","Data":"69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d"} Jul 23 11:55:10.426035 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:10.425989 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" podStartSLOduration=1.4259724870000001 podStartE2EDuration="1.425972487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:55:10.424442158 +0000 UTC m=+1704.406439666" watchObservedRunningTime="2026-07-23 11:55:10.425972487 +0000 UTC m=+1704.407969991" Jul 23 11:55:11.413494 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:11.413457 2564 generic.go:356] "Generic (PLEG): container finished" podID="f339c3c9-12a1-409e-81bd-cb4754b62d0b" containerID="24a8bf7e497d2ac36ba4bfe0a9a4e27fd5abef171a34771d0d98490291c73d48" exitCode=0 Jul 23 11:55:11.413921 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:11.413510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" event={"ID":"f339c3c9-12a1-409e-81bd-cb4754b62d0b","Type":"ContainerDied","Data":"24a8bf7e497d2ac36ba4bfe0a9a4e27fd5abef171a34771d0d98490291c73d48"} Jul 23 11:55:12.549734 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:12.549711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:12.643316 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:12.643292 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca\") pod \"f339c3c9-12a1-409e-81bd-cb4754b62d0b\" (UID: \"f339c3c9-12a1-409e-81bd-cb4754b62d0b\") " Jul 23 11:55:12.643643 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:12.643596 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f339c3c9-12a1-409e-81bd-cb4754b62d0b" (UID: "f339c3c9-12a1-409e-81bd-cb4754b62d0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:55:12.744491 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:12.744463 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f339c3c9-12a1-409e-81bd-cb4754b62d0b-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:13.425901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:13.425875 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" Jul 23 11:55:13.426081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:13.425923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92" event={"ID":"f339c3c9-12a1-409e-81bd-cb4754b62d0b","Type":"ContainerDied","Data":"69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d"} Jul 23 11:55:13.426081 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:13.425972 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b244690339b0daecb893cfb4974e6df16716b702c468765ad2e0c04c39693d" Jul 23 11:55:20.044312 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:20.044280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:24.615496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.615461 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92"] Jul 23 11:55:24.622053 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.622018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:24.622239 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.622038 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4beeb76-hhs92"] Jul 23 11:55:24.757346 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.757307 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:55:24.757623 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.757574 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" containerID="cri-o://5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e" gracePeriod=30 Jul 23 11:55:24.779737 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:24.779696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:25.048905 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:25.048864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:25.529787 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:25.529726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:25.715855 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:25.715817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:25.815980 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:25.815891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:26.543296 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:26.543264 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f339c3c9-12a1-409e-81bd-cb4754b62d0b" path="/var/lib/kubelet/pods/f339c3c9-12a1-409e-81bd-cb4754b62d0b/volumes" Jul 23 11:55:27.191624 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:27.191595 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:30.027397 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.027336 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:30.213062 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213062 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213099 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213164 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213294 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213277 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213301 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blhcs\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs\") pod \"6a37ab87-26f1-4c7b-b305-19994c069c24\" (UID: \"6a37ab87-26f1-4c7b-b305-19994c069c24\") " Jul 23 11:55:30.213593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213355 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:30.213593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213459 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:30.213593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213477 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:30.213593 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213523 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:55:30.213895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213645 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.213895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213662 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37ab87-26f1-4c7b-b305-19994c069c24-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.213895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213673 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.213895 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.213685 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.214276 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.214246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:30.215674 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.215653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs" (OuterVolumeSpecName: "kube-api-access-blhcs") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "kube-api-access-blhcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:55:30.215898 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.215859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:30.216123 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.216106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 11:55:30.216182 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.216128 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a37ab87-26f1-4c7b-b305-19994c069c24" (UID: "6a37ab87-26f1-4c7b-b305-19994c069c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:55:30.314810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.314669 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.314810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.314703 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.314810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.314717 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.314810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.314728 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhcs\" (UniqueName: \"kubernetes.io/projected/6a37ab87-26f1-4c7b-b305-19994c069c24-kube-api-access-blhcs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.314810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.314743 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37ab87-26f1-4c7b-b305-19994c069c24-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:55:30.530521 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.530486 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerID="5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e" exitCode=0 Jul 23 11:55:30.530694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.530544 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" Jul 23 11:55:30.530694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.530571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerDied","Data":"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e"} Jul 23 11:55:30.530694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.530635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp" event={"ID":"6a37ab87-26f1-4c7b-b305-19994c069c24","Type":"ContainerDied","Data":"648d62ea66c1c5fef343134f005bed8e86aa08d2edc8ccd264636d303c198040"} Jul 23 11:55:30.530694 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.530667 2564 scope.go:117] "RemoveContainer" containerID="5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e" Jul 23 11:55:30.540941 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.540922 2564 scope.go:117] "RemoveContainer" containerID="b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3" Jul 23 11:55:30.550719 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.550658 2564 scope.go:117] "RemoveContainer" containerID="5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e" Jul 23 11:55:30.550980 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:55:30.550963 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e\": container with ID starting with 5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e not found: ID does not exist" containerID="5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e" Jul 23 11:55:30.551067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.550998 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e"} err="failed to get container status \"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e\": rpc error: code = NotFound desc = could not find container \"5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e\": container with ID starting with 5722d01351a08c8a3c33fa888d1f7c1bae28ef8a1a492bef97c2335b12482b6e not found: ID does not exist" Jul 23 11:55:30.551067 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.551014 2564 scope.go:117] "RemoveContainer" containerID="b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3" Jul 23 11:55:30.551259 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:55:30.551237 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3\": container with ID starting with b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3 not found: ID does not exist" containerID="b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3" Jul 23 11:55:30.551323 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.551266 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3"} err="failed to get container status \"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3\": rpc error: code = NotFound desc = could not find container \"b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3\": container with ID starting with b6b0ee5a18288f3482ebdb111c50b48e7bc121faa5b4b9ad79039563922100a3 not found: ID does not exist" Jul 23 11:55:30.553072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.553046 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:55:30.557672 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:30.557649 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a4beeb76-openshift-default-679ccc79cf-6dltp"] Jul 23 11:55:32.543148 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:32.543113 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" path="/var/lib/kubelet/pods/6a37ab87-26f1-4c7b-b305-19994c069c24/volumes" Jul 23 11:55:33.159690 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:33.159656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:33.261456 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:33.261406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:33.431609 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:33.431500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:33.526470 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:33.526434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:33.906850 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:33.906813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:55:59.982794 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:55:59.982727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:07.163151 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:07.163095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:07.270083 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:07.270053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:07.371183 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:07.371155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:23.347912 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:23.347878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:30.159180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:30.159147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:30.320579 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:30.320521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:30.442107 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:30.441994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:34.668591 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:34.668541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:34.769658 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:34.769625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:41.039213 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:41.039166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:41.783306 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:41.783272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:46.614329 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:46.614300 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:56:46.618897 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:46.618876 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 11:56:48.816299 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:48.816266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:48.922434 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:48.922397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:49.030850 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:49.030819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:49.141202 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:49.141116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:49.948480 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:49.948413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:50.055145 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:50.055109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:50.148727 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:50.148696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:51.894159 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:51.894126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:57.600506 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:57.600468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:57.671687 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:57.671653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:56:58.291362 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:56:58.291335 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:04.393138 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:04.393101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:04.497218 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:04.497184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:04.665255 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:04.665160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:05.013603 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.013573 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:05.014019 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014005 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.014019 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014020 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.014122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014037 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.014122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014043 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.014122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014052 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f339c3c9-12a1-409e-81bd-cb4754b62d0b" containerName="revoke-keys" Jul 23 11:57:05.014122 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014058 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f339c3c9-12a1-409e-81bd-cb4754b62d0b" containerName="revoke-keys" Jul 23 11:57:05.014263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014133 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f339c3c9-12a1-409e-81bd-cb4754b62d0b" containerName="revoke-keys" Jul 23 11:57:05.014263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014141 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.014263 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.014150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a37ab87-26f1-4c7b-b305-19994c069c24" containerName="istio-proxy" Jul 23 11:57:05.017283 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.017263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.021090 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.021062 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:57:05.021384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.021363 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bd7354bb-openshift-default-dockercfg-6gmhh\"" Jul 23 11:57:05.034999 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.034975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:05.064721 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.064681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:05.175780 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.175724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.175956 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.175790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.175956 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.175871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p248z\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176064 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.175972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176064 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.176030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.176060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.176097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176234 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.176158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.176234 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.176182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.232026 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.231997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:05.277212 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277212 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277431 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277431 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.277673 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p248z\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.278004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.278004 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.277869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.278444 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.278401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.278810 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.278747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.279076 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.279054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.280529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.280510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.280615 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.280508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.289429 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.289405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.290280 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.290236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p248z\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z\") pod \"e2e-mt-bd7354bb-openshift-default-57974875f-jn76t\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.330672 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.330644 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:05.492147 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.492121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:05.494620 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:57:05.494568 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a34d37c_6445_4265_b910_da6fe86546f9.slice/crio-7dfa83a5cf31299671ad7aade1f9e755f21f2213c1cc0487911e848661b37f4d WatchSource:0}: Error finding container 7dfa83a5cf31299671ad7aade1f9e755f21f2213c1cc0487911e848661b37f4d: Status 404 returned error can't find the container with id 7dfa83a5cf31299671ad7aade1f9e755f21f2213c1cc0487911e848661b37f4d Jul 23 11:57:05.497881 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.497847 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:05.497979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.497915 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:05.497979 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:05.497941 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:06.115431 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:06.115382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerStarted","Data":"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d"} Jul 23 11:57:06.115821 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:06.115455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerStarted","Data":"7dfa83a5cf31299671ad7aade1f9e755f21f2213c1cc0487911e848661b37f4d"} Jul 23 11:57:06.151154 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:06.151099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podStartSLOduration=2.151077526 podStartE2EDuration="2.151077526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:57:06.148392617 +0000 UTC m=+1820.130390121" watchObservedRunningTime="2026-07-23 11:57:06.151077526 +0000 UTC m=+1820.133075028" Jul 23 11:57:06.331652 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:06.331619 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:06.336723 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:06.336695 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:07.125258 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:07.125216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:07.126249 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:07.126224 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:08.405597 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:08.405562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:08.497325 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:08.497292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:08.695701 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:08.695627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:08.945549 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:08.945519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:09.489777 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:09.489729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:09.585533 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:09.585501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:12.792865 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:12.792827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:13.471345 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:13.471316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:13.564843 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:13.564814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:13.952112 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:13.952026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:14.621175 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:14.621141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:19.755789 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:19.755672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:19.817414 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:19.817376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:20.123589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.123543 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr"] Jul 23 11:57:20.126797 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.126755 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:20.129589 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.129569 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:57:20.137128 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.137054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr"] Jul 23 11:57:20.216212 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.216177 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a34d37c-6445-4265-b910-da6fe86546f9" containerID="5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d" exitCode=0 Jul 23 11:57:20.216352 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.216249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerDied","Data":"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d"} Jul 23 11:57:20.216619 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.216605 2564 scope.go:117] "RemoveContainer" containerID="5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d" Jul 23 11:57:20.217568 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.217540 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:20.217627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.217613 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:20.217669 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.217647 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:57:20.218585 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.218565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr\" (UID: \"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:20.319744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.319710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr\" (UID: \"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:20.320318 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.320297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr\" (UID: \"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:20.438470 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.438373 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:20.591487 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:20.591460 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr"] Jul 23 11:57:20.592694 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:57:20.592671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b03c07_1580_4123_ad7d_caa0ca2d9d6d.slice/crio-b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a WatchSource:0}: Error finding container b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a: Status 404 returned error can't find the container with id b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a Jul 23 11:57:21.224516 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.224477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" event={"ID":"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d","Type":"ContainerStarted","Data":"ced706fba02d3f58930ca1aadbc0745bb97ffa9dab682c530045acc1ed8c3afe"} Jul 23 11:57:21.225027 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.224536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" event={"ID":"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d","Type":"ContainerStarted","Data":"b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a"} Jul 23 11:57:21.226648 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.226625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerStarted","Data":"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14"} Jul 23 11:57:21.241384 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.241343 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" podStartSLOduration=1.241329987 podStartE2EDuration="1.241329987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:57:21.240399263 +0000 UTC m=+1835.222396780" watchObservedRunningTime="2026-07-23 11:57:21.241329987 +0000 UTC m=+1835.223327490" Jul 23 11:57:21.331233 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.331192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:21.332724 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.332692 2564 patch_prober.go:28] interesting pod/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 23 11:57:21.332878 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:21.332753 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 23 11:57:22.125680 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:22.125643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:22.234467 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:22.234437 2564 generic.go:356] "Generic (PLEG): container finished" podID="a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" containerID="ced706fba02d3f58930ca1aadbc0745bb97ffa9dab682c530045acc1ed8c3afe" exitCode=0 Jul 23 11:57:22.234861 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:22.234525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" event={"ID":"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d","Type":"ContainerDied","Data":"ced706fba02d3f58930ca1aadbc0745bb97ffa9dab682c530045acc1ed8c3afe"} Jul 23 11:57:22.331388 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:22.331345 2564 patch_prober.go:28] interesting pod/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 23 11:57:22.331584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:22.331405 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 23 11:57:23.331688 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.331658 2564 patch_prober.go:28] interesting pod/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 23 11:57:23.332137 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.331714 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 23 11:57:23.422686 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.422660 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:23.449103 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.449079 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca\") pod \"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d\" (UID: \"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d\") " Jul 23 11:57:23.449392 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.449371 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" (UID: "a7b03c07-1580-4123-ad7d-caa0ca2d9d6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:57:23.550639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:23.550544 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:24.248180 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:24.248145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" event={"ID":"a7b03c07-1580-4123-ad7d-caa0ca2d9d6d","Type":"ContainerDied","Data":"b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a"} Jul 23 11:57:24.248357 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:24.248188 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr" Jul 23 11:57:24.248357 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:24.248195 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ea822d5b43801858ce41bebdc477777fd578998b94f63f087ab59b10222e6a" Jul 23 11:57:24.331438 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:24.331402 2564 patch_prober.go:28] interesting pod/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 23 11:57:24.331618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:24.331457 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 23 11:57:25.335169 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:25.335138 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:25.336100 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:25.336083 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:30.626594 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:30.626560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:35.204287 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:35.204252 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr"] Jul 23 11:57:35.206803 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:35.206755 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7354bb-pfczr"] Jul 23 11:57:35.242075 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:35.242039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:35.771753 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:35.771712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:36.544339 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:36.544304 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" path="/var/lib/kubelet/pods/a7b03c07-1580-4123-ad7d-caa0ca2d9d6d/volumes" Jul 23 11:57:41.229631 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.229587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:41.291539 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.291494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:41.485601 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.485513 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:41.485836 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.485806 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" containerID="cri-o://032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14" gracePeriod=30 Jul 23 11:57:41.508218 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.508182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:41.757082 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.756984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:41.891993 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.891956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:41.990270 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:41.990231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:42.105584 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:42.105481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:45.027704 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:45.027671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:46.740947 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.740922 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:46.781806 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.781952 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.781952 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.781952 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781887 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.781952 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.782179 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.782179 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.781982 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.782179 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782028 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:57:46.782179 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782054 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p248z\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.782380 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782218 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:57:46.782380 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:57:46.782469 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782440 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34d37c-6445-4265-b910-da6fe86546f9-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.782469 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782459 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.782574 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782472 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.782876 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.782852 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:57:46.784301 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.784276 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:57:46.784636 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.784600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:57:46.784904 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.784881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z" (OuterVolumeSpecName: "kube-api-access-p248z") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "kube-api-access-p248z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:57:46.784993 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.784940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 11:57:46.882984 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.882953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket\") pod \"1a34d37c-6445-4265-b910-da6fe86546f9\" (UID: \"1a34d37c-6445-4265-b910-da6fe86546f9\") " Jul 23 11:57:46.883165 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883152 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p248z\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-kube-api-access-p248z\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.883210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883166 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.883210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883177 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34d37c-6445-4265-b910-da6fe86546f9-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.883210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883184 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34d37c-6445-4265-b910-da6fe86546f9-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.883210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883193 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:46.883210 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.883197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a34d37c-6445-4265-b910-da6fe86546f9" (UID: "1a34d37c-6445-4265-b910-da6fe86546f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:57:46.983804 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:46.983757 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34d37c-6445-4265-b910-da6fe86546f9-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:57:47.388450 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.388410 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a34d37c-6445-4265-b910-da6fe86546f9" containerID="032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14" exitCode=0 Jul 23 11:57:47.388627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.388457 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" Jul 23 11:57:47.388627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.388460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerDied","Data":"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14"} Jul 23 11:57:47.388627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.388584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t" event={"ID":"1a34d37c-6445-4265-b910-da6fe86546f9","Type":"ContainerDied","Data":"7dfa83a5cf31299671ad7aade1f9e755f21f2213c1cc0487911e848661b37f4d"} Jul 23 11:57:47.388627 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.388608 2564 scope.go:117] "RemoveContainer" containerID="032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14" Jul 23 11:57:47.399127 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.399107 2564 scope.go:117] "RemoveContainer" containerID="5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d" Jul 23 11:57:47.408676 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.408658 2564 scope.go:117] "RemoveContainer" containerID="032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14" Jul 23 11:57:47.408986 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:57:47.408966 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14\": container with ID starting with 032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14 not found: ID does not exist" containerID="032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14" Jul 23 11:57:47.409072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.408994 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14"} err="failed to get container status \"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14\": rpc error: code = NotFound desc = could not find container \"032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14\": container with ID starting with 032e665c4ef2517a82c34171d0f939a11008099e19e44474e867ccc771f20c14 not found: ID does not exist" Jul 23 11:57:47.409072 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.409012 2564 scope.go:117] "RemoveContainer" containerID="5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d" Jul 23 11:57:47.409262 ip-10-0-128-30 kubenswrapper[2564]: E0723 11:57:47.409242 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d\": container with ID starting with 5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d not found: ID does not exist" containerID="5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d" Jul 23 11:57:47.409321 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.409267 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d"} err="failed to get container status \"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d\": rpc error: code = NotFound desc = could not find container \"5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d\": container with ID starting with 5322f67a61ee6997fc1ba0312e94fc13efcbd6e31351389a1ea6b7fb7655d18d not found: ID does not exist" Jul 23 11:57:47.410145 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.410124 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:47.413795 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.413752 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bd7354bb-openshift-default-57974875f-jn76t"] Jul 23 11:57:47.842016 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.841950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:47.959445 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:47.959411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:48.157489 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:48.157364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:48.265467 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:48.265431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:48.543066 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:48.543032 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" path="/var/lib/kubelet/pods/1a34d37c-6445-4265-b910-da6fe86546f9/volumes" Jul 23 11:57:53.985592 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:53.985535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:57:54.045946 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:57:54.045906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:00.909873 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:00.909838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:01.348048 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348015 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 11:58:01.348437 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348425 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:01.348496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348439 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:01.348496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" containerName="revoke-keys" Jul 23 11:58:01.348496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" containerName="revoke-keys" Jul 23 11:58:01.348496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348465 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:01.348496 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:01.348679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348545 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:01.348679 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.348553 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b03c07-1580-4123-ad7d-caa0ca2d9d6d" containerName="revoke-keys" Jul 23 11:58:01.353383 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.353365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.357406 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.357386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b7dc3a03-openshift-default-dockercfg-gxxwf\"" Jul 23 11:58:01.359788 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.359748 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:58:01.372070 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.372034 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 11:58:01.407542 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzxv\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.407815 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.408138 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.408138 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.407862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.429830 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.429792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:01.508960 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.508922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.508973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzxv\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509167 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509459 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509618 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509727 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.509843 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.509825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.511511 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.511493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.511817 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.511788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.519253 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.519218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.519748 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.519723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzxv\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv\") pod \"e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.664374 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.664277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:01.820096 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.820072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 11:58:01.822145 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:58:01.822118 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3888b0a_8baa_45b1_9351_1c71047351e1.slice/crio-cc2085e1dc24903fb3256c355d6e5a1a182b3c0ec16e7dd29dc79bb713a0118d WatchSource:0}: Error finding container cc2085e1dc24903fb3256c355d6e5a1a182b3c0ec16e7dd29dc79bb713a0118d: Status 404 returned error can't find the container with id cc2085e1dc24903fb3256c355d6e5a1a182b3c0ec16e7dd29dc79bb713a0118d Jul 23 11:58:01.824863 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.824827 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:01.824967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.824899 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:01.824967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.824930 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:01.850520 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:01.850499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:02.481163 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:02.481124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerStarted","Data":"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2"} Jul 23 11:58:02.481598 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:02.481182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerStarted","Data":"cc2085e1dc24903fb3256c355d6e5a1a182b3c0ec16e7dd29dc79bb713a0118d"} Jul 23 11:58:02.513197 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:02.511096 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podStartSLOduration=1.511075891 podStartE2EDuration="1.511075891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:58:02.505426507 +0000 UTC m=+1876.487424010" watchObservedRunningTime="2026-07-23 11:58:02.511075891 +0000 UTC m=+1876.493073395" Jul 23 11:58:02.664575 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:02.664538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:02.669601 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:02.669565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:03.486913 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:03.486879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:03.487859 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:03.487841 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:04.447408 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:04.447377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:04.552840 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:04.552810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:04.646164 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:04.646131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:04.745260 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:04.745233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:05.097109 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:05.097027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:05.203084 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:05.203056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:08.925134 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:08.925057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:17.573740 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.573699 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerID="8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2" exitCode=0 Jul 23 11:58:17.574268 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.573794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerDied","Data":"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2"} Jul 23 11:58:17.574345 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.574287 2564 scope.go:117] "RemoveContainer" containerID="8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2" Jul 23 11:58:17.575473 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.575435 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:17.575541 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.575526 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:17.575607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:17.575580 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:58:18.487529 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:18.487486 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:18.588447 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:18.588413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerStarted","Data":"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a"} Jul 23 11:58:18.664776 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:18.664726 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:18.665744 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:18.665716 2564 patch_prober.go:28] interesting pod/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 23 11:58:18.665885 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:18.665790 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 23 11:58:19.665266 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:19.665231 2564 patch_prober.go:28] interesting pod/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 23 11:58:19.665639 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:19.665300 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 23 11:58:20.664684 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:20.664654 2564 patch_prober.go:28] interesting pod/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 23 11:58:20.664935 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:20.664712 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 23 11:58:21.664607 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:21.664578 2564 patch_prober.go:28] interesting pod/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 23 11:58:21.665021 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:21.664627 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 23 11:58:22.668596 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:22.668566 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:22.669034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:22.668843 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:22.669567 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:22.669548 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 11:58:23.465281 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:23.465245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:58:52.645092 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.645059 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j"] Jul 23 11:58:52.645591 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.645511 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a34d37c-6445-4265-b910-da6fe86546f9" containerName="istio-proxy" Jul 23 11:58:52.649901 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.649884 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:52.652720 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.652695 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 11:58:52.655930 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.655905 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j"] Jul 23 11:58:52.788985 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.788954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j\" (UID: \"8c767b0f-ad0e-47c2-a96b-b9deab352a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:52.889573 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.889537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j\" (UID: \"8c767b0f-ad0e-47c2-a96b-b9deab352a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:52.890331 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.890302 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j\" (UID: \"8c767b0f-ad0e-47c2-a96b-b9deab352a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:52.962181 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:52.962102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:53.106059 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:53.106028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j"] Jul 23 11:58:53.107299 ip-10-0-128-30 kubenswrapper[2564]: W0723 11:58:53.107267 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c767b0f_ad0e_47c2_a96b_b9deab352a5e.slice/crio-00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f WatchSource:0}: Error finding container 00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f: Status 404 returned error can't find the container with id 00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f Jul 23 11:58:53.809859 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:53.809818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" event={"ID":"8c767b0f-ad0e-47c2-a96b-b9deab352a5e","Type":"ContainerStarted","Data":"e87e8525bdf05b686cfde45b32351634e53e3067a4a6f70acd306f4db8b1c342"} Jul 23 11:58:53.810234 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:53.809888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" event={"ID":"8c767b0f-ad0e-47c2-a96b-b9deab352a5e","Type":"ContainerStarted","Data":"00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f"} Jul 23 11:58:53.827077 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:53.827030 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" podStartSLOduration=1.827015901 podStartE2EDuration="1.827015901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:58:53.824942047 +0000 UTC m=+1927.806939552" watchObservedRunningTime="2026-07-23 11:58:53.827015901 +0000 UTC m=+1927.809013403" Jul 23 11:58:54.817967 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:54.817932 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c767b0f-ad0e-47c2-a96b-b9deab352a5e" containerID="e87e8525bdf05b686cfde45b32351634e53e3067a4a6f70acd306f4db8b1c342" exitCode=0 Jul 23 11:58:54.818337 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:54.818014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" event={"ID":"8c767b0f-ad0e-47c2-a96b-b9deab352a5e","Type":"ContainerDied","Data":"e87e8525bdf05b686cfde45b32351634e53e3067a4a6f70acd306f4db8b1c342"} Jul 23 11:58:55.971788 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:55.971748 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:56.120115 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.120029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca\") pod \"8c767b0f-ad0e-47c2-a96b-b9deab352a5e\" (UID: \"8c767b0f-ad0e-47c2-a96b-b9deab352a5e\") " Jul 23 11:58:56.120405 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.120377 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c767b0f-ad0e-47c2-a96b-b9deab352a5e" (UID: "8c767b0f-ad0e-47c2-a96b-b9deab352a5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:58:56.221034 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.221006 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b0f-ad0e-47c2-a96b-b9deab352a5e-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 11:58:56.832052 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.831961 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" Jul 23 11:58:56.832204 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.831960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j" event={"ID":"8c767b0f-ad0e-47c2-a96b-b9deab352a5e","Type":"ContainerDied","Data":"00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f"} Jul 23 11:58:56.832204 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:58:56.832089 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00b5194c833b4017706b86000f42ac286f393b60b0e1907c140c57c283ac663f" Jul 23 11:59:08.165080 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:08.165042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:59:12.743221 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:12.743188 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j"] Jul 23 11:59:12.745094 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:12.745067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:59:12.745814 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:12.745791 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7dc3a03-f4r4j"] Jul 23 11:59:14.543251 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:14.543219 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c767b0f-ad0e-47c2-a96b-b9deab352a5e" path="/var/lib/kubelet/pods/8c767b0f-ad0e-47c2-a96b-b9deab352a5e/volumes" Jul 23 11:59:14.949843 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:14.949747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:59:19.980849 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:19.980813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:59:46.748605 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:46.748573 2564 scope.go:117] "RemoveContainer" containerID="ccee51b4a384a1273a4f07777ee289e9ab1da0ad2634a5d3c1fe62ac892db330" Jul 23 11:59:46.758135 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:46.758117 2564 scope.go:117] "RemoveContainer" containerID="3bc8bb0ef197857e3f0e013ea14fdb156083a7fab24eaa3cb32cbc0fd61d4b44" Jul 23 11:59:56.378388 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:56.378348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 11:59:57.028252 ip-10-0-128-30 kubenswrapper[2564]: I0723 11:59:57.028217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:00.139278 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.139242 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84"] Jul 23 12:00:00.139677 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.139658 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c767b0f-ad0e-47c2-a96b-b9deab352a5e" containerName="revoke-keys" Jul 23 12:00:00.139677 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.139670 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c767b0f-ad0e-47c2-a96b-b9deab352a5e" containerName="revoke-keys" Jul 23 12:00:00.139783 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.139752 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c767b0f-ad0e-47c2-a96b-b9deab352a5e" containerName="revoke-keys" Jul 23 12:00:00.142916 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.142895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" Jul 23 12:00:00.145484 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.145461 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:00:00.149840 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.149455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84"] Jul 23 12:00:00.157026 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.157007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" Jul 23 12:00:00.310236 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:00.310203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84"] Jul 23 12:00:00.311187 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:00:00.311164 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada63923_294a_4b2e_9bab_fd32f6e22013.slice/crio-c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec WatchSource:0}: Error finding container c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec: Status 404 returned error can't find the container with id c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec Jul 23 12:00:01.228001 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:01.227967 2564 generic.go:356] "Generic (PLEG): container finished" podID="ada63923-294a-4b2e-9bab-fd32f6e22013" containerID="0916006a37ed6998970155b915fca8888b75781082cbc4d8dfd56294abfffe67" exitCode=0 Jul 23 12:00:01.228367 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:01.228042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" event={"ID":"ada63923-294a-4b2e-9bab-fd32f6e22013","Type":"ContainerDied","Data":"0916006a37ed6998970155b915fca8888b75781082cbc4d8dfd56294abfffe67"} Jul 23 12:00:01.228367 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:01.228100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" event={"ID":"ada63923-294a-4b2e-9bab-fd32f6e22013","Type":"ContainerStarted","Data":"c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec"} Jul 23 12:00:02.369128 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:02.369107 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" Jul 23 12:00:02.937518 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:02.937435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:03.241840 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:03.241804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" event={"ID":"ada63923-294a-4b2e-9bab-fd32f6e22013","Type":"ContainerDied","Data":"c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec"} Jul 23 12:00:03.241997 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:03.241867 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c224cd69f3e6412c8c23dcbd2d7695f9be384ed0ec046a8968074075359a55ec" Jul 23 12:00:03.241997 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:03.241932 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-lxj84" Jul 23 12:00:07.079010 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:07.078978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:07.181171 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:07.181122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:08.251711 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.251673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:08.352154 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.352101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:08.583992 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.583909 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 12:00:08.584139 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.584117 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" containerID="cri-o://c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a" gracePeriod=30 Jul 23 12:00:08.587572 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.587547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:08.787957 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:08.787929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:09.875052 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:09.875018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:09.972957 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:09.972923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:10.072131 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:10.072100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:10.816655 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:10.816620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:10.904646 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:10.904617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:11.003821 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:11.003782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:12.429917 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:12.429885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:13.857267 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.857207 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 12:00:13.918491 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918491 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918496 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918644 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbzxv\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.918724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.919053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo\") pod \"f3888b0a-8baa-45b1-9351-1c71047351e1\" (UID: \"f3888b0a-8baa-45b1-9351-1c71047351e1\") " Jul 23 12:00:13.919053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.918965 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:00:13.919163 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.919129 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:00:13.919223 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.919156 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:00:13.919223 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.919174 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:13.919223 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.919175 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:00:13.919657 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.919629 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:00:13.920939 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.920913 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:00:13.921050 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.921024 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv" (OuterVolumeSpecName: "kube-api-access-zbzxv") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "kube-api-access-zbzxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:00:13.921356 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.921328 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:00:13.921466 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:13.921448 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3888b0a-8baa-45b1-9351-1c71047351e1" (UID: "f3888b0a-8baa-45b1-9351-1c71047351e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:00:14.020681 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020641 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020681 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020681 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020695 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020708 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3888b0a-8baa-45b1-9351-1c71047351e1-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020721 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020736 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzxv\" (UniqueName: \"kubernetes.io/projected/f3888b0a-8baa-45b1-9351-1c71047351e1-kube-api-access-zbzxv\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020751 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3888b0a-8baa-45b1-9351-1c71047351e1-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.020900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.020786 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3888b0a-8baa-45b1-9351-1c71047351e1-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:00:14.308288 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.308248 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerID="c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a" exitCode=0 Jul 23 12:00:14.308440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.308295 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" Jul 23 12:00:14.308440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.308345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerDied","Data":"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a"} Jul 23 12:00:14.308440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.308402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh" event={"ID":"f3888b0a-8baa-45b1-9351-1c71047351e1","Type":"ContainerDied","Data":"cc2085e1dc24903fb3256c355d6e5a1a182b3c0ec16e7dd29dc79bb713a0118d"} Jul 23 12:00:14.308440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.308429 2564 scope.go:117] "RemoveContainer" containerID="c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a" Jul 23 12:00:14.319359 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.319340 2564 scope.go:117] "RemoveContainer" containerID="8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2" Jul 23 12:00:14.328073 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.328056 2564 scope.go:117] "RemoveContainer" containerID="c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a" Jul 23 12:00:14.328390 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:00:14.328331 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a\": container with ID starting with c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a not found: ID does not exist" containerID="c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a" Jul 23 12:00:14.328390 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.328362 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a"} err="failed to get container status \"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a\": rpc error: code = NotFound desc = could not find container \"c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a\": container with ID starting with c0091a79562560ae13650159a772eec04550227bc0ac6374c4e3b7a4e7864c2a not found: ID does not exist" Jul 23 12:00:14.328390 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.328387 2564 scope.go:117] "RemoveContainer" containerID="8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2" Jul 23 12:00:14.328731 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:00:14.328704 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2\": container with ID starting with 8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2 not found: ID does not exist" containerID="8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2" Jul 23 12:00:14.328831 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.328741 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2"} err="failed to get container status \"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2\": rpc error: code = NotFound desc = could not find container \"8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2\": container with ID starting with 8fcf453ba51fff4e1236d7d1079ec8233efd56e4ded4a5903593d574597ee7b2 not found: ID does not exist" Jul 23 12:00:14.331957 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.331936 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 12:00:14.337228 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.337204 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b7dc3a03-openshift-default-55b69c785f-sjjxh"] Jul 23 12:00:14.543217 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:14.543186 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" path="/var/lib/kubelet/pods/f3888b0a-8baa-45b1-9351-1c71047351e1/volumes" Jul 23 12:00:18.412171 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:18.412136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:18.512041 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:18.512008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:19.102468 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:19.102441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:24.985227 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:24.985189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:25.087331 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:25.087300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:25.197647 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:25.197613 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:35.738432 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:35.738392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:36.145853 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:36.145747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:00:46.786484 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:46.786445 2564 scope.go:117] "RemoveContainer" containerID="d1f68eb50727bd7f075c628aa6612372e99b1a3df0aa68b998321791d1aa6e2e" Jul 23 12:00:57.090665 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:00:57.090627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:03.364695 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:03.364660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:03.466182 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:03.466147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:03.727523 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:03.727489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:04.018127 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018053 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:01:04.018533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018509 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.018533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018530 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.018533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada63923-294a-4b2e-9bab-fd32f6e22013" containerName="cleanup" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018546 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada63923-294a-4b2e-9bab-fd32f6e22013" containerName="cleanup" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018562 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018568 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018640 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018649 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada63923-294a-4b2e-9bab-fd32f6e22013" containerName="cleanup" Jul 23 12:01:04.018886 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.018656 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3888b0a-8baa-45b1-9351-1c71047351e1" containerName="istio-proxy" Jul 23 12:01:04.022084 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.022065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.025042 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.025022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f8adc6-openshift-default-dockercfg-7dltp\"" Jul 23 12:01:04.026210 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.026191 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 12:01:04.033719 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.033695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:01:04.047898 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.047878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:04.087785 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrjm\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.087972 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.087931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.088254 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.088013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.088254 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.088067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.188918 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.188885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189105 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.188928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189105 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.188959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189105 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189105 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189105 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrjm\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189463 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.189740 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.189719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.190094 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.190072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.191976 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.191951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.192275 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.192256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.197401 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.197379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrjm\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.197564 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.197542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token\") pod \"e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.221745 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.221717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:04.333443 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.333351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:04.478885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.478852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:01:04.479985 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:01:04.479947 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f1729e_af13_4a55_8ce7_9af9764084e6.slice/crio-8f8fcbe0ca70a67b6a1340a9e88382e0d670c3f742693cda9169fa09696f6492 WatchSource:0}: Error finding container 8f8fcbe0ca70a67b6a1340a9e88382e0d670c3f742693cda9169fa09696f6492: Status 404 returned error can't find the container with id 8f8fcbe0ca70a67b6a1340a9e88382e0d670c3f742693cda9169fa09696f6492 Jul 23 12:01:04.482304 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.482273 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:04.482384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.482336 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:04.482384 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.482367 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:04.626939 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.626845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerStarted","Data":"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80"} Jul 23 12:01:04.626939 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.626896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerStarted","Data":"8f8fcbe0ca70a67b6a1340a9e88382e0d670c3f742693cda9169fa09696f6492"} Jul 23 12:01:04.650891 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:04.650843 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podStartSLOduration=1.650828041 podStartE2EDuration="1.650828041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:01:04.647138892 +0000 UTC m=+2058.629136397" watchObservedRunningTime="2026-07-23 12:01:04.650828041 +0000 UTC m=+2058.632825547" Jul 23 12:01:05.334080 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:05.334043 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:05.339018 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:05.338992 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:05.633948 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:05.633866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:05.634913 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:05.634893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:06.664413 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:06.664332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:06.763490 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:06.763459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:06.870254 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:06.870225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:07.150668 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:07.150640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:07.478214 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:07.478185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:07.578312 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:07.578286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:11.117770 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.117726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:11.458218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.458134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:11.767056 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.767027 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:01:11.770912 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.770886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.773559 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.773536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-591d5c-openshift-default-dockercfg-kglpd\"" Jul 23 12:01:11.786861 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.786837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:01:11.814910 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.814882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:11.869920 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.869890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870085 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.869990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870085 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870181 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870181 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870258 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870258 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870258 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqf8\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.870358 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.870297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.901585 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.901560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:11.970872 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.970841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.970920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.970949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971275 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971275 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971275 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqf8\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971438 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971529 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971593 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971593 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.971717 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.973990 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.971791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.973990 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.973931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.974170 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.974011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.980222 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.980179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqf8\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:11.980586 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:11.980565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token\") pod \"e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:12.082869 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.082787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:12.232103 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.232074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:01:12.233005 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:01:12.232982 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dea6b06_2c3f_4927_a146_09e948a8dda5.slice/crio-3079ee529304c815f6d7c09131a34ef36e3659514258670fce1df1ba78f5eade WatchSource:0}: Error finding container 3079ee529304c815f6d7c09131a34ef36e3659514258670fce1df1ba78f5eade: Status 404 returned error can't find the container with id 3079ee529304c815f6d7c09131a34ef36e3659514258670fce1df1ba78f5eade Jul 23 12:01:12.235385 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.235350 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:12.235459 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.235438 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:12.235508 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.235492 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:12.681418 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.681381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerStarted","Data":"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf"} Jul 23 12:01:12.681593 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.681441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerStarted","Data":"3079ee529304c815f6d7c09131a34ef36e3659514258670fce1df1ba78f5eade"} Jul 23 12:01:12.708863 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:12.708816 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podStartSLOduration=1.708799989 podStartE2EDuration="1.708799989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:01:12.705385904 +0000 UTC m=+2066.687383417" watchObservedRunningTime="2026-07-23 12:01:12.708799989 +0000 UTC m=+2066.690797486" Jul 23 12:01:13.083608 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:13.083561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:13.088941 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:13.088912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:13.687603 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:13.687561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:13.688714 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:13.688693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:14.728255 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:14.728223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:14.821929 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:14.821888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:14.941296 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:14.941265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:15.197366 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:15.197285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:15.410379 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:15.410350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:15.511170 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:15.511142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:17.863920 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:17.863888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:19.166911 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:19.166866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:21.675941 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:21.675887 2564 patch_prober.go:28] interesting pod/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:01:21.676430 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:21.675963 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:01:23.750101 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.750064 2564 generic.go:356] "Generic (PLEG): container finished" podID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerID="cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80" exitCode=0 Jul 23 12:01:23.750535 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.750137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerDied","Data":"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80"} Jul 23 12:01:23.750581 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.750540 2564 scope.go:117] "RemoveContainer" containerID="cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80" Jul 23 12:01:23.751564 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.751533 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:23.751646 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.751616 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:23.751689 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:23.751653 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:24.713786 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:24.713738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:24.757686 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:24.757645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerStarted","Data":"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f"} Jul 23 12:01:25.334755 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:25.334719 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:25.334962 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:25.334868 2564 patch_prober.go:28] interesting pod/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 23 12:01:25.334962 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:25.334908 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 23 12:01:26.333875 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:26.333833 2564 patch_prober.go:28] interesting pod/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 23 12:01:26.334265 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:26.333895 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 23 12:01:27.334605 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:27.334565 2564 patch_prober.go:28] interesting pod/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 23 12:01:27.335051 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:27.334619 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 23 12:01:28.334368 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:28.334331 2564 patch_prober.go:28] interesting pod/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 23 12:01:28.334623 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:28.334401 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 23 12:01:28.405786 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:28.405422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:29.338604 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.338571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:29.339084 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.339056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:29.339702 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.339683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:01:29.406573 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.406505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:29.511745 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.511713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:29.728930 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.728879 2564 patch_prober.go:28] interesting pod/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:01:29.729147 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:29.728948 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:01:31.806846 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.806811 2564 generic.go:356] "Generic (PLEG): container finished" podID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerID="7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf" exitCode=0 Jul 23 12:01:31.807445 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.806885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerDied","Data":"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf"} Jul 23 12:01:31.807445 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.807256 2564 scope.go:117] "RemoveContainer" containerID="7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf" Jul 23 12:01:31.808326 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.808295 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:31.808396 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.808381 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:31.808436 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:31.808421 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:01:32.815726 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:32.815693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerStarted","Data":"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a"} Jul 23 12:01:33.083741 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:33.082965 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:33.083741 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:33.083352 2564 patch_prober.go:28] interesting pod/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 23 12:01:33.083741 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:33.083399 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 23 12:01:34.083583 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:34.083529 2564 patch_prober.go:28] interesting pod/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 23 12:01:34.084116 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:34.083625 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 23 12:01:35.084367 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:35.084330 2564 patch_prober.go:28] interesting pod/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 23 12:01:35.084970 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:35.084939 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 23 12:01:36.084527 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:36.084070 2564 patch_prober.go:28] interesting pod/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 23 12:01:36.084527 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:36.084153 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 23 12:01:36.594106 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:36.594067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:36.695515 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:36.695485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:01:37.087541 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:37.087510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:37.087983 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:37.087792 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:37.088692 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:37.088667 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:01:46.662673 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:46.662641 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:01:46.669073 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:46.669052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:01:46.814237 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:46.814206 2564 scope.go:117] "RemoveContainer" containerID="24a8bf7e497d2ac36ba4bfe0a9a4e27fd5abef171a34771d0d98490291c73d48" Jul 23 12:01:48.678069 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:01:48.678022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:28.561227 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:28.561198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:29.554227 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.554191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:29.665150 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.665111 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5"] Jul 23 12:02:29.667884 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.667866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:29.670388 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.670367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:02:29.677274 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.676905 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5"] Jul 23 12:02:29.730326 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.730291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5\" (UID: \"7331d2cb-91d2-4ee4-9a67-b7982763e832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:29.831740 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.831638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5\" (UID: \"7331d2cb-91d2-4ee4-9a67-b7982763e832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:29.832343 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.832321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5\" (UID: \"7331d2cb-91d2-4ee4-9a67-b7982763e832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:29.981879 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:29.981843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:30.132265 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:30.132235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5"] Jul 23 12:02:30.134564 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:02:30.134540 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7331d2cb_91d2_4ee4_9a67_b7982763e832.slice/crio-c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052 WatchSource:0}: Error finding container c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052: Status 404 returned error can't find the container with id c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052 Jul 23 12:02:30.162270 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:30.162237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" event={"ID":"7331d2cb-91d2-4ee4-9a67-b7982763e832","Type":"ContainerStarted","Data":"c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052"} Jul 23 12:02:30.209748 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:30.209717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:30.308926 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:30.308897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:31.169231 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:31.169147 2564 generic.go:356] "Generic (PLEG): container finished" podID="7331d2cb-91d2-4ee4-9a67-b7982763e832" containerID="43e5366b3515338b8878d31c347fb1a88a2f85949e479f9b6f2d3e9e566ac90b" exitCode=0 Jul 23 12:02:31.169572 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:31.169233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" event={"ID":"7331d2cb-91d2-4ee4-9a67-b7982763e832","Type":"ContainerDied","Data":"43e5366b3515338b8878d31c347fb1a88a2f85949e479f9b6f2d3e9e566ac90b"} Jul 23 12:02:32.322379 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:32.322356 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:32.455980 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:32.455894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca\") pod \"7331d2cb-91d2-4ee4-9a67-b7982763e832\" (UID: \"7331d2cb-91d2-4ee4-9a67-b7982763e832\") " Jul 23 12:02:32.456234 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:32.456209 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7331d2cb-91d2-4ee4-9a67-b7982763e832" (UID: "7331d2cb-91d2-4ee4-9a67-b7982763e832"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:02:32.557445 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:32.557421 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7331d2cb-91d2-4ee4-9a67-b7982763e832-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:33.183383 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:33.183350 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" Jul 23 12:02:33.183383 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:33.183374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5" event={"ID":"7331d2cb-91d2-4ee4-9a67-b7982763e832","Type":"ContainerDied","Data":"c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052"} Jul 23 12:02:33.183658 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:33.183455 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b9add180984169b7300d8f2f09d8c2de992127cbabb03ee253afb7205f3052" Jul 23 12:02:40.110296 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:40.110259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:44.741733 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:44.741695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:44.756330 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:44.756298 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5"] Jul 23 12:02:44.758727 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:44.758705 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8adc6-bvfv5"] Jul 23 12:02:46.028826 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:46.028785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:46.544615 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:46.544580 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7331d2cb-91d2-4ee4-9a67-b7982763e832" path="/var/lib/kubelet/pods/7331d2cb-91d2-4ee4-9a67-b7982763e832/volumes" Jul 23 12:02:50.989906 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:50.989865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:51.090583 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.090546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:51.249791 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.249682 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:02:51.249976 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.249951 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" containerID="cri-o://3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f" gracePeriod=30 Jul 23 12:02:51.280367 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.280327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:51.501813 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.501699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:51.624808 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.624736 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9"] Jul 23 12:02:51.625234 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.625207 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7331d2cb-91d2-4ee4-9a67-b7982763e832" containerName="revoke-keys" Jul 23 12:02:51.625234 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.625225 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7331d2cb-91d2-4ee4-9a67-b7982763e832" containerName="revoke-keys" Jul 23 12:02:51.625434 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.625306 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7331d2cb-91d2-4ee4-9a67-b7982763e832" containerName="revoke-keys" Jul 23 12:02:51.627897 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.627879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:51.630230 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.630212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-smx5c\"" Jul 23 12:02:51.637673 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.637648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9"] Jul 23 12:02:51.641422 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.641324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9\" (UID: \"a1795101-fba1-48e9-a7d9-1e262ac7b74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:51.742049 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.742014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9\" (UID: \"a1795101-fba1-48e9-a7d9-1e262ac7b74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:51.742674 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.742655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9\" (UID: \"a1795101-fba1-48e9-a7d9-1e262ac7b74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:51.940288 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:51.940191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:52.088316 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:52.088289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9"] Jul 23 12:02:52.090804 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:02:52.090747 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1795101_fba1_48e9_a7d9_1e262ac7b74b.slice/crio-487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a WatchSource:0}: Error finding container 487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a: Status 404 returned error can't find the container with id 487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a Jul 23 12:02:52.196270 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:52.196189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:52.296705 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:52.296680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:02:52.314148 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:52.314116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" event={"ID":"a1795101-fba1-48e9-a7d9-1e262ac7b74b","Type":"ContainerStarted","Data":"487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a"} Jul 23 12:02:53.321412 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:53.321374 2564 generic.go:356] "Generic (PLEG): container finished" podID="a1795101-fba1-48e9-a7d9-1e262ac7b74b" containerID="bf4d565f5a4b186304b90c0df31d95217bc769d2cf1119209e1f2b7e083b1a6f" exitCode=0 Jul 23 12:02:53.321878 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:53.321464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" event={"ID":"a1795101-fba1-48e9-a7d9-1e262ac7b74b","Type":"ContainerDied","Data":"bf4d565f5a4b186304b90c0df31d95217bc769d2cf1119209e1f2b7e083b1a6f"} Jul 23 12:02:54.470139 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:54.470116 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:54.563516 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:54.563489 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca\") pod \"a1795101-fba1-48e9-a7d9-1e262ac7b74b\" (UID: \"a1795101-fba1-48e9-a7d9-1e262ac7b74b\") " Jul 23 12:02:54.563909 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:54.563889 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1795101-fba1-48e9-a7d9-1e262ac7b74b" (UID: "a1795101-fba1-48e9-a7d9-1e262ac7b74b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:02:54.664304 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:54.664223 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1795101-fba1-48e9-a7d9-1e262ac7b74b-maas-api-service-ca\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:55.335235 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:55.335196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" event={"ID":"a1795101-fba1-48e9-a7d9-1e262ac7b74b","Type":"ContainerDied","Data":"487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a"} Jul 23 12:02:55.335423 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:55.335259 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487e1476fa58903355b12e8c435309a9b340da65a1a867e6da9a6c914754e49a" Jul 23 12:02:55.335423 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:55.335207 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9" Jul 23 12:02:56.513468 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.513411 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:02:56.583373 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583381 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583421 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583452 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldrjm\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token\") pod \"b5f1729e-af13-4a55-8ce7-9af9764084e6\" (UID: \"b5f1729e-af13-4a55-8ce7-9af9764084e6\") " Jul 23 12:02:56.583883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583687 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:56.583883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:56.584098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.583924 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:02:56.584098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.584040 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.584098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.584060 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.584098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.584074 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f1729e-af13-4a55-8ce7-9af9764084e6-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.584098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.584072 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:56.585468 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.585433 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:56.585945 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.585916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm" (OuterVolumeSpecName: "kube-api-access-ldrjm") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "kube-api-access-ldrjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:02:56.586026 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.585919 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:02:56.586101 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.586085 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:02:56.586438 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.586425 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5f1729e-af13-4a55-8ce7-9af9764084e6" (UID: "b5f1729e-af13-4a55-8ce7-9af9764084e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:02:56.684849 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684820 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.684849 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684846 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.685014 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684858 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.685014 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684871 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f1729e-af13-4a55-8ce7-9af9764084e6-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.685014 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684883 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrjm\" (UniqueName: \"kubernetes.io/projected/b5f1729e-af13-4a55-8ce7-9af9764084e6-kube-api-access-ldrjm\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:56.685014 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:56.684894 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f1729e-af13-4a55-8ce7-9af9764084e6-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:02:57.348853 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.348823 2564 generic.go:356] "Generic (PLEG): container finished" podID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerID="3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f" exitCode=0 Jul 23 12:02:57.349053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.348879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" Jul 23 12:02:57.349053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.348908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerDied","Data":"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f"} Jul 23 12:02:57.349053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.348961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t" event={"ID":"b5f1729e-af13-4a55-8ce7-9af9764084e6","Type":"ContainerDied","Data":"8f8fcbe0ca70a67b6a1340a9e88382e0d670c3f742693cda9169fa09696f6492"} Jul 23 12:02:57.349053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.348985 2564 scope.go:117] "RemoveContainer" containerID="3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f" Jul 23 12:02:57.358980 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.358964 2564 scope.go:117] "RemoveContainer" containerID="cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80" Jul 23 12:02:57.367818 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.367640 2564 scope.go:117] "RemoveContainer" containerID="3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f" Jul 23 12:02:57.367938 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:02:57.367917 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f\": container with ID starting with 3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f not found: ID does not exist" containerID="3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f" Jul 23 12:02:57.367980 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.367947 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f"} err="failed to get container status \"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f\": rpc error: code = NotFound desc = could not find container \"3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f\": container with ID starting with 3e265693e21fc34a4ea1c7192f55dab54dae27056167a89e47b41d096f2bb19f not found: ID does not exist" Jul 23 12:02:57.367980 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.367963 2564 scope.go:117] "RemoveContainer" containerID="cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80" Jul 23 12:02:57.368210 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:02:57.368194 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80\": container with ID starting with cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80 not found: ID does not exist" containerID="cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80" Jul 23 12:02:57.368256 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.368218 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80"} err="failed to get container status \"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80\": rpc error: code = NotFound desc = could not find container \"cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80\": container with ID starting with cd4ab012b5c99ef41a55757551398e97b6963e696ca399f383d986b94deb8b80 not found: ID does not exist" Jul 23 12:02:57.374281 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.374257 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:02:57.376732 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:57.376711 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f8adc6-openshift-default-b756b56f7-vhz2t"] Jul 23 12:02:58.543616 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:02:58.543585 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" path="/var/lib/kubelet/pods/b5f1729e-af13-4a55-8ce7-9af9764084e6/volumes" Jul 23 12:03:02.889724 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:02.889684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:06.717612 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:06.717576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:06.726180 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:06.726147 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9"] Jul 23 12:03:06.729004 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:06.728984 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-591d5c-p7pp9"] Jul 23 12:03:07.104584 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:07.104553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:08.544029 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:08.543997 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1795101-fba1-48e9-a7d9-1e262ac7b74b" path="/var/lib/kubelet/pods/a1795101-fba1-48e9-a7d9-1e262ac7b74b/volumes" Jul 23 12:03:13.217620 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.217584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:13.282262 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.282228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:13.486214 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.486179 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:03:13.486500 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.486472 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" containerID="cri-o://476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a" gracePeriod=30 Jul 23 12:03:13.515910 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.515872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:13.722392 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:13.722349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:14.063061 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.063005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:14.356073 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.355991 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:03:14.356423 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356415 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.356464 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356426 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.356464 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356452 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1795101-fba1-48e9-a7d9-1e262ac7b74b" containerName="revoke-keys" Jul 23 12:03:14.356464 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356458 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1795101-fba1-48e9-a7d9-1e262ac7b74b" containerName="revoke-keys" Jul 23 12:03:14.356562 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356465 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.356562 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356471 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.356562 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356529 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1795101-fba1-48e9-a7d9-1e262ac7b74b" containerName="revoke-keys" Jul 23 12:03:14.356562 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356538 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.356562 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.356546 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f1729e-af13-4a55-8ce7-9af9764084e6" containerName="istio-proxy" Jul 23 12:03:14.361626 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.361604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.364653 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.364636 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-43f746-openshift-default-dockercfg-pxglw\"" Jul 23 12:03:14.374869 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.374841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:03:14.418649 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.418616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:14.451097 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451097 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tmq\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451318 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451318 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451318 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451441 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451441 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451441 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.451441 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.451428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552045 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tmq\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552701 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.552815 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.552792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.553134 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.553112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.554568 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.554548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.554852 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.554835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.561853 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.561832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.562005 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.561988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tmq\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq\") pod \"e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.672879 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.672786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:14.736539 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.736447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:14.829413 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.829379 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:03:14.831132 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:03:14.831107 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1824108_12e7_42ac_aa23_95751240673b.slice/crio-2d1d9def8355cd3b2ddd01b09d318bf0402a0fea009e3b41e0a8384860edbe72 WatchSource:0}: Error finding container 2d1d9def8355cd3b2ddd01b09d318bf0402a0fea009e3b41e0a8384860edbe72: Status 404 returned error can't find the container with id 2d1d9def8355cd3b2ddd01b09d318bf0402a0fea009e3b41e0a8384860edbe72 Jul 23 12:03:14.833699 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.833658 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:14.833794 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.833723 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:14.833794 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:14.833757 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:15.460719 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:15.460681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerStarted","Data":"bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b"} Jul 23 12:03:15.461115 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:15.460735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerStarted","Data":"2d1d9def8355cd3b2ddd01b09d318bf0402a0fea009e3b41e0a8384860edbe72"} Jul 23 12:03:15.480589 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:15.480542 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podStartSLOduration=1.480524309 podStartE2EDuration="1.480524309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:03:15.478888118 +0000 UTC m=+2189.460885634" watchObservedRunningTime="2026-07-23 12:03:15.480524309 +0000 UTC m=+2189.462521814" Jul 23 12:03:15.673829 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:15.673795 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:15.679605 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:15.679579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:16.466636 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:16.466609 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:16.467435 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:16.467414 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:17.432363 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:17.432332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:17.532365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:17.532334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:17.633298 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:17.633247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:17.732706 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:17.732672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:18.104645 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.104565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:18.245561 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.245526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:18.757671 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.757607 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:03:18.889286 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889353 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbqf8\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889806 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889806 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889521 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889806 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889558 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token\") pod \"0dea6b06-2c3f-4927-a146-09e948a8dda5\" (UID: \"0dea6b06-2c3f-4927-a146-09e948a8dda5\") " Jul 23 12:03:18.889806 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889719 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:03:18.890009 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:03:18.890009 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889894 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:03:18.890009 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.889919 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.890267 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.890234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:03:18.890667 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.890638 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:03:18.891992 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.891968 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8" (OuterVolumeSpecName: "kube-api-access-dbqf8") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "kube-api-access-dbqf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:03:18.892227 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.892206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:03:18.892398 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.892374 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:03:18.892467 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.892397 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0dea6b06-2c3f-4927-a146-09e948a8dda5" (UID: "0dea6b06-2c3f-4927-a146-09e948a8dda5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:03:18.990600 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990567 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbqf8\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-kube-api-access-dbqf8\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990600 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990598 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990600 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990609 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dea6b06-2c3f-4927-a146-09e948a8dda5-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990860 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990616 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990860 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990625 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990860 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990632 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990860 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990639 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dea6b06-2c3f-4927-a146-09e948a8dda5-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:18.990860 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:18.990646 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dea6b06-2c3f-4927-a146-09e948a8dda5-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:03:19.488192 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.488156 2564 generic.go:356] "Generic (PLEG): container finished" podID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerID="476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a" exitCode=0 Jul 23 12:03:19.488371 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.488213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" Jul 23 12:03:19.488371 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.488226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerDied","Data":"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a"} Jul 23 12:03:19.488371 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.488269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4" event={"ID":"0dea6b06-2c3f-4927-a146-09e948a8dda5","Type":"ContainerDied","Data":"3079ee529304c815f6d7c09131a34ef36e3659514258670fce1df1ba78f5eade"} Jul 23 12:03:19.488371 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.488290 2564 scope.go:117] "RemoveContainer" containerID="476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a" Jul 23 12:03:19.498364 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.498341 2564 scope.go:117] "RemoveContainer" containerID="7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf" Jul 23 12:03:19.507008 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.506991 2564 scope.go:117] "RemoveContainer" containerID="476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a" Jul 23 12:03:19.507242 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:03:19.507223 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a\": container with ID starting with 476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a not found: ID does not exist" containerID="476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a" Jul 23 12:03:19.507302 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.507249 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a"} err="failed to get container status \"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a\": rpc error: code = NotFound desc = could not find container \"476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a\": container with ID starting with 476ac710e41587a18aa91b4eb7e67aca1bca1b2a61a6fd5cd8e99c5ee9405a8a not found: ID does not exist" Jul 23 12:03:19.507302 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.507267 2564 scope.go:117] "RemoveContainer" containerID="7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf" Jul 23 12:03:19.507501 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:03:19.507486 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf\": container with ID starting with 7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf not found: ID does not exist" containerID="7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf" Jul 23 12:03:19.507545 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.507504 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf"} err="failed to get container status \"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf\": rpc error: code = NotFound desc = could not find container \"7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf\": container with ID starting with 7c3cfbabc823bf16f18443df8325ef2b368df2957de496be06b7552e976a6faf not found: ID does not exist" Jul 23 12:03:19.516220 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.516197 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:03:19.518586 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:19.518563 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-591d5c-openshift-default-6b9779bbcc-lsnh4"] Jul 23 12:03:20.543663 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:20.543630 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" path="/var/lib/kubelet/pods/0dea6b06-2c3f-4927-a146-09e948a8dda5/volumes" Jul 23 12:03:21.952264 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:21.952232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:22.447140 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.447098 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:22.739169 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.739132 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:03:22.740273 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.740246 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:03:22.740437 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.740425 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:03:22.740941 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.740917 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:03:22.741012 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.740950 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:03:22.745822 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.745797 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.748584 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.748563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f1f99f-openshift-default-dockercfg-v6ccc\"" Jul 23 12:03:22.749668 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.749648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:03:22.824912 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.824877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825074 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.824937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825074 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825180 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825180 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825180 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825180 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825332 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4xm\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.825332 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.825280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926208 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926387 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926387 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926519 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926519 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926519 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926519 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4xm\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926716 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926716 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926716 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.926894 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.926788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.927053 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.927033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.927325 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.927307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.927378 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.927313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.928639 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.928617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.928885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.928865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.934627 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.934608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:22.934950 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:22.934932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4xm\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm\") pod \"e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:23.057522 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.057437 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:23.204503 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.204475 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:03:23.205650 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:03:23.205625 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283a0e79_d39f_46a9_889a_b7e6f051d8e1.slice/crio-43017435748ddf82d657b60b6e191af789e54db292cbc3210cb1ddd65d31d7aa WatchSource:0}: Error finding container 43017435748ddf82d657b60b6e191af789e54db292cbc3210cb1ddd65d31d7aa: Status 404 returned error can't find the container with id 43017435748ddf82d657b60b6e191af789e54db292cbc3210cb1ddd65d31d7aa Jul 23 12:03:23.207900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.207869 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:23.207996 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.207945 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:23.208064 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.207993 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:23.490100 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.490068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:23.515275 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.515243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerStarted","Data":"dcc0245de06c84e9558891138ec110354ce26c5a8bd48c6b4a3969849d87d176"} Jul 23 12:03:23.515443 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.515298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerStarted","Data":"43017435748ddf82d657b60b6e191af789e54db292cbc3210cb1ddd65d31d7aa"} Jul 23 12:03:23.536435 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.536389 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podStartSLOduration=1.536373082 podStartE2EDuration="1.536373082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:03:23.534441065 +0000 UTC m=+2197.516438569" watchObservedRunningTime="2026-07-23 12:03:23.536373082 +0000 UTC m=+2197.518370642" Jul 23 12:03:23.590773 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:23.590741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:24.058429 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:24.058396 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:24.063440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:24.063413 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:24.521304 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:24.521277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:24.522172 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:24.522151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:25.558544 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:25.558503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:25.655730 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:25.655701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:25.789723 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:25.789688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:26.220647 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:26.220620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:26.346876 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:26.346844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:26.447500 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:26.447440 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:29.966672 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:29.966638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:32.509944 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:32.509899 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:03:32.510337 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:32.509974 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:03:33.693084 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:03:33.693043 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1824108_12e7_42ac_aa23_95751240673b.slice/crio-conmon-bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b.scope\": RecentStats: unable to find data in memory cache]" Jul 23 12:03:34.592703 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.592662 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1824108-12e7-42ac-aa23-95751240673b" containerID="bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b" exitCode=0 Jul 23 12:03:34.592965 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.592733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerDied","Data":"bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b"} Jul 23 12:03:34.593205 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.593189 2564 scope.go:117] "RemoveContainer" containerID="bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b" Jul 23 12:03:34.594253 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.594223 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:34.594356 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.594299 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:34.594356 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:34.594329 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:35.628580 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:35.628535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerStarted","Data":"fcb3c89edd94e4e8da532d52740e2f23c094c56a0bb2653177f709d6012df6b5"} Jul 23 12:03:35.683230 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:35.683194 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:35.683615 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:35.683591 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 23 12:03:35.683888 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:35.683859 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 23 12:03:36.685221 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:36.685180 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 23 12:03:36.685870 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:36.685833 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 23 12:03:37.673447 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:37.673409 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 23 12:03:37.673752 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:37.673723 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 23 12:03:38.674089 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:38.674042 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 23 12:03:38.674599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:38.674111 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 23 12:03:38.865017 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:38.864971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:38.966955 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:38.966858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:39.677552 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:39.677517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:39.678017 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:39.677869 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:39.678688 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:39.678671 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:03:40.562958 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:40.562916 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:03:40.563168 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:40.562979 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:03:42.675209 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.675109 2564 generic.go:356] "Generic (PLEG): container finished" podID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerID="dcc0245de06c84e9558891138ec110354ce26c5a8bd48c6b4a3969849d87d176" exitCode=0 Jul 23 12:03:42.675209 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.675181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerDied","Data":"dcc0245de06c84e9558891138ec110354ce26c5a8bd48c6b4a3969849d87d176"} Jul 23 12:03:42.675677 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.675591 2564 scope.go:117] "RemoveContainer" containerID="dcc0245de06c84e9558891138ec110354ce26c5a8bd48c6b4a3969849d87d176" Jul 23 12:03:42.676722 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.676688 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:42.676835 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.676795 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:42.676877 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:42.676846 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:03:43.683757 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:43.683721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerStarted","Data":"897ae5808e7c1886b6a8863a7f8bc043512adc624f19531d891ea955ede83d69"} Jul 23 12:03:44.058413 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:44.058376 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:44.058606 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:44.058584 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 23 12:03:44.058653 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:44.058632 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 23 12:03:45.058827 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:45.058788 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 23 12:03:45.059195 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:45.058846 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 23 12:03:46.054611 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:46.054574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:46.058412 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:46.058380 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 23 12:03:46.058529 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:46.058431 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 23 12:03:46.846717 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:46.846684 2564 scope.go:117] "RemoveContainer" containerID="ced706fba02d3f58930ca1aadbc0745bb97ffa9dab682c530045acc1ed8c3afe" Jul 23 12:03:47.057893 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:47.057855 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 23 12:03:47.058061 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:47.057909 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 23 12:03:47.242820 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:47.242787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:47.345153 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:47.345121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:48.061251 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:48.061218 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:48.061716 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:48.061510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:48.062392 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:48.062371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:03:53.296943 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:53.296908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:58.184462 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:58.184427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:03:58.572835 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:03:58.572806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:04.365481 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.365439 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:04.465575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.465539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:04.631185 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.631108 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:04:04.631341 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.631320 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" containerID="cri-o://fcb3c89edd94e4e8da532d52740e2f23c094c56a0bb2653177f709d6012df6b5" gracePeriod=30 Jul 23 12:04:04.660634 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.660597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:04.892238 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:04.892148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:09.679371 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.679343 2564 patch_prober.go:28] interesting pod/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 23 12:04:09.679841 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.679407 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 23 12:04:09.849883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.849850 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1824108-12e7-42ac-aa23-95751240673b" containerID="fcb3c89edd94e4e8da532d52740e2f23c094c56a0bb2653177f709d6012df6b5" exitCode=0 Jul 23 12:04:09.850047 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.849935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerDied","Data":"fcb3c89edd94e4e8da532d52740e2f23c094c56a0bb2653177f709d6012df6b5"} Jul 23 12:04:09.850047 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.850008 2564 scope.go:117] "RemoveContainer" containerID="bc81ff0581cf6fb1360f6cd48c55e3840dd000496c7c1e945f874126639c3c0b" Jul 23 12:04:09.895245 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.895190 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:04:09.909267 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909245 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909382 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909284 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909382 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909382 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909320 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8tmq\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909382 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909382 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909562 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert\") pod \"d1824108-12e7-42ac-aa23-95751240673b\" (UID: \"d1824108-12e7-42ac-aa23-95751240673b\") " Jul 23 12:04:09.909632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909547 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:09.909632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909585 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:09.909922 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909813 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:09.909922 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909916 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:09.910029 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909934 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:09.910029 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.909948 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:09.910323 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.910286 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:09.910744 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.910717 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:09.911936 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.911900 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq" (OuterVolumeSpecName: "kube-api-access-q8tmq") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "kube-api-access-q8tmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:09.912036 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.912008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:09.912211 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.912185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:04:09.912211 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:09.912195 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1824108-12e7-42ac-aa23-95751240673b" (UID: "d1824108-12e7-42ac-aa23-95751240673b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:10.010864 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010830 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.010864 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010855 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1824108-12e7-42ac-aa23-95751240673b-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.010864 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010865 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.010864 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010873 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tmq\" (UniqueName: \"kubernetes.io/projected/d1824108-12e7-42ac-aa23-95751240673b-kube-api-access-q8tmq\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.011144 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010881 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1824108-12e7-42ac-aa23-95751240673b-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.011144 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.010891 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1824108-12e7-42ac-aa23-95751240673b-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:10.856925 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.856838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" event={"ID":"d1824108-12e7-42ac-aa23-95751240673b","Type":"ContainerDied","Data":"2d1d9def8355cd3b2ddd01b09d318bf0402a0fea009e3b41e0a8384860edbe72"} Jul 23 12:04:10.856925 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.856857 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb" Jul 23 12:04:10.856925 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.856897 2564 scope.go:117] "RemoveContainer" containerID="fcb3c89edd94e4e8da532d52740e2f23c094c56a0bb2653177f709d6012df6b5" Jul 23 12:04:10.884462 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.878621 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:04:10.887751 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:10.887724 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-43f746-openshift-default-57c575ddb7-l67nb"] Jul 23 12:04:12.543583 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:12.543536 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1824108-12e7-42ac-aa23-95751240673b" path="/var/lib/kubelet/pods/d1824108-12e7-42ac-aa23-95751240673b/volumes" Jul 23 12:04:20.776499 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:20.776464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:25.106499 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:25.106466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:25.495016 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:25.494973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:31.580059 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:31.580026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:31.678265 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:31.678230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:31.828123 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:31.828089 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:04:31.828405 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:31.828374 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" containerID="cri-o://897ae5808e7c1886b6a8863a7f8bc043512adc624f19531d891ea955ede83d69" gracePeriod=30 Jul 23 12:04:31.869952 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:31.869855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:32.117268 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.117227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:32.461885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.461843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:32.770057 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770031 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:04:32.770454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770442 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:04:32.770498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dea6b06-2c3f-4927-a146-09e948a8dda5" containerName="istio-proxy" Jul 23 12:04:32.770498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770466 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.770498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770472 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.770498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770481 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.770498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770488 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.770663 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770554 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.770663 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.770563 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1824108-12e7-42ac-aa23-95751240673b" containerName="istio-proxy" Jul 23 12:04:32.773062 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.773033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.776027 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.775993 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7cc2e5-openshift-default-dockercfg-wjk5l\"" Jul 23 12:04:32.788911 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.788888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:04:32.808255 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.808216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:32.928100 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928285 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928285 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928285 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrxr\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928285 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928474 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928474 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928474 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.928474 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.928401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:32.934750 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:32.934728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:33.029121 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029121 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029121 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrxr\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029365 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029664 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029664 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.029664 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.029653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.030097 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.030066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.030097 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.030084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.031661 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.031641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.031774 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.031687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.042269 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.042246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.042408 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.042391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrxr\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr\") pod \"e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.064265 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.064240 2564 patch_prober.go:28] interesting pod/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:04:33.064369 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.064301 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:04:33.085662 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.085636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:33.226809 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.226779 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:04:33.228145 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:04:33.228118 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512b52fa_5713_497c_804e_4afdefadbd04.slice/crio-d5fcd8ae41825b75a73d212421d94fc4fc40589b477244039f08b5c9b5fabefc WatchSource:0}: Error finding container d5fcd8ae41825b75a73d212421d94fc4fc40589b477244039f08b5c9b5fabefc: Status 404 returned error can't find the container with id d5fcd8ae41825b75a73d212421d94fc4fc40589b477244039f08b5c9b5fabefc Jul 23 12:04:33.230141 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.230109 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:33.230249 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.230188 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:33.230249 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.230233 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:33.997223 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.997184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerStarted","Data":"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0"} Jul 23 12:04:33.997628 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:33.997237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerStarted","Data":"d5fcd8ae41825b75a73d212421d94fc4fc40589b477244039f08b5c9b5fabefc"} Jul 23 12:04:34.022380 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:34.022333 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podStartSLOduration=2.022318421 podStartE2EDuration="2.022318421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:04:34.019755389 +0000 UTC m=+2268.001752891" watchObservedRunningTime="2026-07-23 12:04:34.022318421 +0000 UTC m=+2268.004315924" Jul 23 12:04:34.086715 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:34.086683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:34.091805 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:34.091784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:35.005488 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:35.005452 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:35.006513 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:35.006489 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:36.004985 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:36.004955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:36.109885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:36.109851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:36.200362 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:36.200332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:36.719512 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:36.719477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:36.819479 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:36.819444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:37.022577 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.022546 2564 generic.go:356] "Generic (PLEG): container finished" podID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerID="897ae5808e7c1886b6a8863a7f8bc043512adc624f19531d891ea955ede83d69" exitCode=0 Jul 23 12:04:37.022755 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.022618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerDied","Data":"897ae5808e7c1886b6a8863a7f8bc043512adc624f19531d891ea955ede83d69"} Jul 23 12:04:37.022755 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.022681 2564 scope.go:117] "RemoveContainer" containerID="dcc0245de06c84e9558891138ec110354ce26c5a8bd48c6b4a3969849d87d176" Jul 23 12:04:37.100778 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.100738 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:04:37.270902 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.270841 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.270902 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.270894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.270916 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.270940 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4xm\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.270980 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271153 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271326 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:37.272317 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.272121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy\") pod \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\" (UID: \"283a0e79-d39f-46a9-889a-b7e6f051d8e1\") " Jul 23 12:04:37.272920 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.272521 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.272920 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.272541 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.272920 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.272554 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.273984 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.273660 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:37.274479 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.271437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:37.275547 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.275522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:37.276313 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.276277 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:04:37.276543 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.276513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm" (OuterVolumeSpecName: "kube-api-access-2g4xm") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "kube-api-access-2g4xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:37.276821 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.276800 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "283a0e79-d39f-46a9-889a-b7e6f051d8e1" (UID: "283a0e79-d39f-46a9-889a-b7e6f051d8e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:37.373512 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373480 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.373512 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373515 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.373784 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373528 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.373784 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373536 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.373784 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373546 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4xm\" (UniqueName: \"kubernetes.io/projected/283a0e79-d39f-46a9-889a-b7e6f051d8e1-kube-api-access-2g4xm\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:37.373784 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:37.373554 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/283a0e79-d39f-46a9-889a-b7e6f051d8e1-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:04:38.030816 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.030785 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" Jul 23 12:04:38.030990 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.030787 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw" event={"ID":"283a0e79-d39f-46a9-889a-b7e6f051d8e1","Type":"ContainerDied","Data":"43017435748ddf82d657b60b6e191af789e54db292cbc3210cb1ddd65d31d7aa"} Jul 23 12:04:38.030990 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.030922 2564 scope.go:117] "RemoveContainer" containerID="897ae5808e7c1886b6a8863a7f8bc043512adc624f19531d891ea955ede83d69" Jul 23 12:04:38.054378 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.054341 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:04:38.060779 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.060736 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f1f99f-openshift-default-75546dbd67-bl4bw"] Jul 23 12:04:38.550416 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:38.550371 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" path="/var/lib/kubelet/pods/283a0e79-d39f-46a9-889a-b7e6f051d8e1/volumes" Jul 23 12:04:40.500427 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:40.500383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:40.847542 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:40.847443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:41.156014 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.155931 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:04:41.156362 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156349 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.156431 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156362 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.156431 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156377 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.156431 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156383 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.156595 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156446 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.156595 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.156455 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="283a0e79-d39f-46a9-889a-b7e6f051d8e1" containerName="istio-proxy" Jul 23 12:04:41.159733 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.159710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.164627 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.164592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-77f87c-openshift-default-dockercfg-spf25\"" Jul 23 12:04:41.173026 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.173003 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:04:41.197649 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.197623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:41.308340 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308340 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308533 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308706 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwm9\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308706 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.308706 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.308633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwm9\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409575 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409921 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409921 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.409921 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.409872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.410069 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.410013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.410069 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.410019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.411945 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.411922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.412071 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.412036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.417307 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.417281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.417420 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.417350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwm9\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9\") pod \"e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.423108 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.423082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:41.469943 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.469920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:41.613131 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.613104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:04:41.615081 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:04:41.615053 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65b8456_66f3_4cf5_886f_a80d05f8b766.slice/crio-60fe96093177926fcac9680298e4266339b5286841baa0989fc09af46ea230a1 WatchSource:0}: Error finding container 60fe96093177926fcac9680298e4266339b5286841baa0989fc09af46ea230a1: Status 404 returned error can't find the container with id 60fe96093177926fcac9680298e4266339b5286841baa0989fc09af46ea230a1 Jul 23 12:04:41.617608 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.617576 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:41.617720 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.617653 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:41.617720 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:41.617700 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:42.056800 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:42.056748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerStarted","Data":"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a"} Jul 23 12:04:42.056970 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:42.056817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerStarted","Data":"60fe96093177926fcac9680298e4266339b5286841baa0989fc09af46ea230a1"} Jul 23 12:04:42.077906 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:42.077850 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podStartSLOduration=1.077831962 podStartE2EDuration="1.077831962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:04:42.07639749 +0000 UTC m=+2276.058394996" watchObservedRunningTime="2026-07-23 12:04:42.077831962 +0000 UTC m=+2276.059829466" Jul 23 12:04:42.470567 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:42.470536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:42.476267 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:42.476237 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:43.062498 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:43.062467 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:43.063521 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:43.063503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:04:44.099312 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.099278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:44.215174 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.215138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:44.299323 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.299293 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:44.609888 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.609859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:44.784883 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.784847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:44.886191 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:44.886099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:48.746552 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:48.746516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:51.047004 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:51.046969 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:04:51.047493 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:51.047032 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:04:52.135199 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.135163 2564 generic.go:356] "Generic (PLEG): container finished" podID="512b52fa-5713-497c-804e-4afdefadbd04" containerID="d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0" exitCode=0 Jul 23 12:04:52.135717 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.135247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerDied","Data":"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0"} Jul 23 12:04:52.135836 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.135753 2564 scope.go:117] "RemoveContainer" containerID="d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0" Jul 23 12:04:52.137114 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.137077 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:52.137214 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.137152 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:52.137214 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:52.137190 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:04:53.143495 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:53.143462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerStarted","Data":"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac"} Jul 23 12:04:54.086487 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:54.086448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:54.086892 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:54.086851 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 23 12:04:54.087114 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:54.087085 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 23 12:04:55.086576 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:55.086538 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 23 12:04:55.087189 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:55.087153 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 23 12:04:56.087544 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:56.087077 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 23 12:04:56.087544 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:56.087142 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 23 12:04:57.086412 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:57.086375 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 23 12:04:57.086753 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:57.086707 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 23 12:04:57.686529 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:57.686475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:57.788029 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:57.787983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:04:58.099140 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:58.097734 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:58.099140 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:58.098986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:04:58.100808 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:04:58.100787 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:05:01.213723 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.213623 2564 generic.go:356] "Generic (PLEG): container finished" podID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerID="d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a" exitCode=0 Jul 23 12:05:01.214244 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.213709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerDied","Data":"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a"} Jul 23 12:05:01.214244 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.214222 2564 scope.go:117] "RemoveContainer" containerID="d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a" Jul 23 12:05:01.215239 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.215213 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:05:01.215301 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.215278 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:05:01.215337 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:01.215310 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 12:05:02.221208 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:02.221177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerStarted","Data":"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4"} Jul 23 12:05:02.470323 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:02.470289 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:05:02.470547 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:02.470526 2564 patch_prober.go:28] interesting pod/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Jul 23 12:05:02.470635 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:02.470577 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Jul 23 12:05:03.470585 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:03.470550 2564 patch_prober.go:28] interesting pod/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Jul 23 12:05:03.471009 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:03.470602 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Jul 23 12:05:04.471184 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:04.471153 2564 patch_prober.go:28] interesting pod/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Jul 23 12:05:04.471589 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:04.471204 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Jul 23 12:05:05.470941 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:05.470905 2564 patch_prober.go:28] interesting pod/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Jul 23 12:05:05.471147 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:05.470967 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Jul 23 12:05:05.611785 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:05.611714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:05.712659 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:05.712626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:06.473565 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:06.473538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:05:06.473959 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:06.473937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:05:06.474591 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:06.474570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:05:07.784699 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:07.784662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:07.885045 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:07.885008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:12.174894 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:12.174861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:25.647214 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:25.647173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:32.084280 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:32.084240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:38.944028 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:38.943989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:39.004775 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:39.004717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:05:46.932739 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:46.932704 2564 scope.go:117] "RemoveContainer" containerID="e87e8525bdf05b686cfde45b32351634e53e3067a4a6f70acd306f4db8b1c342" Jul 23 12:05:52.943155 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:05:52.943122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:06:46.708797 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:06:46.708746 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:06:46.715544 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:06:46.715525 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:07:38.079424 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:07:38.079392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:07:39.172352 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:07:39.172313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:07:56.069619 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:07:56.069586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:08:34.026260 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:08:34.026215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:08:46.975783 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:08:46.975727 2564 scope.go:117] "RemoveContainer" containerID="43e5366b3515338b8878d31c347fb1a88a2f85949e479f9b6f2d3e9e566ac90b" Jul 23 12:09:20.135242 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:20.135212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:09:21.069456 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:21.069425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:09:29.862510 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:29.862469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:09:46.998745 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:46.998710 2564 scope.go:117] "RemoveContainer" containerID="bf4d565f5a4b186304b90c0df31d95217bc769d2cf1119209e1f2b7e083b1a6f" Jul 23 12:09:49.959326 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:49.959283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:09:58.609931 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:58.609901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:09:58.768287 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:09:58.768255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:06.598271 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:06.598188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:06.701330 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:06.701301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:08.585449 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:08.585395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:08.683442 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:08.683405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:26.296997 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:26.296950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:48.482237 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:48.482207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:49.381935 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:49.381895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:50.360983 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.360949 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624660656"] Jul 23 12:10:50.364430 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.364414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:50.366882 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.366860 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bchvj\"" Jul 23 12:10:50.371689 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.371666 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624660656"] Jul 23 12:10:50.376151 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.376129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28c9\" (UniqueName: \"kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9\") pod \"test-curl-7635-139826624660656\" (UID: \"f6d3a35f-433d-43ed-9ae7-138a457e9a5a\") " pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:50.476739 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.476703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28c9\" (UniqueName: \"kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9\") pod \"test-curl-7635-139826624660656\" (UID: \"f6d3a35f-433d-43ed-9ae7-138a457e9a5a\") " pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:50.487066 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.487034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28c9\" (UniqueName: \"kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9\") pod \"test-curl-7635-139826624660656\" (UID: \"f6d3a35f-433d-43ed-9ae7-138a457e9a5a\") " pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:50.675641 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.675546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:50.815135 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.815111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624660656"] Jul 23 12:10:50.817298 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:10:50.817267 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6d3a35f_433d_43ed_9ae7_138a457e9a5a.slice/crio-d60c72897fe52ebb17cd1a63e2edb5a0cdf554aefea20115d7875f9e5f8e7dff WatchSource:0}: Error finding container d60c72897fe52ebb17cd1a63e2edb5a0cdf554aefea20115d7875f9e5f8e7dff: Status 404 returned error can't find the container with id d60c72897fe52ebb17cd1a63e2edb5a0cdf554aefea20115d7875f9e5f8e7dff Jul 23 12:10:50.818540 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:50.818523 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 12:10:51.360071 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:51.360029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624660656" event={"ID":"f6d3a35f-433d-43ed-9ae7-138a457e9a5a","Type":"ContainerStarted","Data":"d60c72897fe52ebb17cd1a63e2edb5a0cdf554aefea20115d7875f9e5f8e7dff"} Jul 23 12:10:52.367771 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:52.367737 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" containerID="ceb13eb79cffc4238d83f9381557be4e4b0c1a4530a99de06b5fa7d94c1c2786" exitCode=0 Jul 23 12:10:52.368236 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:52.367830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624660656" event={"ID":"f6d3a35f-433d-43ed-9ae7-138a457e9a5a","Type":"ContainerDied","Data":"ceb13eb79cffc4238d83f9381557be4e4b0c1a4530a99de06b5fa7d94c1c2786"} Jul 23 12:10:52.740183 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:52.740125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:10:53.516801 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.516754 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:53.539055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.539028 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624660656_f6d3a35f-433d-43ed-9ae7-138a457e9a5a/test-curl-7635-139826624660656/0.log" Jul 23 12:10:53.563497 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.563456 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624660656"] Jul 23 12:10:53.569626 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.569586 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624660656"] Jul 23 12:10:53.610020 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.609987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28c9\" (UniqueName: \"kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9\") pod \"f6d3a35f-433d-43ed-9ae7-138a457e9a5a\" (UID: \"f6d3a35f-433d-43ed-9ae7-138a457e9a5a\") " Jul 23 12:10:53.612218 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.612191 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9" (OuterVolumeSpecName: "kube-api-access-n28c9") pod "f6d3a35f-433d-43ed-9ae7-138a457e9a5a" (UID: "f6d3a35f-433d-43ed-9ae7-138a457e9a5a"). InnerVolumeSpecName "kube-api-access-n28c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:10:53.693097 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.693015 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624662448"] Jul 23 12:10:53.693452 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.693438 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" containerName="test-curl-7635-139826624660656" Jul 23 12:10:53.693511 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.693453 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" containerName="test-curl-7635-139826624660656" Jul 23 12:10:53.693544 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.693516 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" containerName="test-curl-7635-139826624660656" Jul 23 12:10:53.696599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.696583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:53.704348 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.703878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624662448"] Jul 23 12:10:53.711147 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.711121 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28c9\" (UniqueName: \"kubernetes.io/projected/f6d3a35f-433d-43ed-9ae7-138a457e9a5a-kube-api-access-n28c9\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:10:53.812227 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.812193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjxt\" (UniqueName: \"kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt\") pod \"test-curl-7635-139826624662448\" (UID: \"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f\") " pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:53.912743 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.912705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjxt\" (UniqueName: \"kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt\") pod \"test-curl-7635-139826624662448\" (UID: \"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f\") " pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:53.922078 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:53.922051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjxt\" (UniqueName: \"kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt\") pod \"test-curl-7635-139826624662448\" (UID: \"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f\") " pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:54.007900 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.007866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:54.153313 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.153284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624662448"] Jul 23 12:10:54.154947 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:10:54.154925 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf86cc9f0_6ebd_4435_9dac_a923de7e1b0f.slice/crio-b0aa95708c36b3f1e3753ac1c13862234829690d5b828bc73f41a98bf630d509 WatchSource:0}: Error finding container b0aa95708c36b3f1e3753ac1c13862234829690d5b828bc73f41a98bf630d509: Status 404 returned error can't find the container with id b0aa95708c36b3f1e3753ac1c13862234829690d5b828bc73f41a98bf630d509 Jul 23 12:10:54.382247 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.382157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624662448" event={"ID":"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f","Type":"ContainerStarted","Data":"b0aa95708c36b3f1e3753ac1c13862234829690d5b828bc73f41a98bf630d509"} Jul 23 12:10:54.383387 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.383369 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:54.383504 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.383418 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60c72897fe52ebb17cd1a63e2edb5a0cdf554aefea20115d7875f9e5f8e7dff" Jul 23 12:10:54.394400 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:10:54.394368 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624660656\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" pod="opendatahub/test-curl-7635-139826624660656" Jul 23 12:10:54.543510 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:54.543482 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d3a35f-433d-43ed-9ae7-138a457e9a5a" path="/var/lib/kubelet/pods/f6d3a35f-433d-43ed-9ae7-138a457e9a5a/volumes" Jul 23 12:10:55.391199 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:55.391152 2564 generic.go:356] "Generic (PLEG): container finished" podID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" containerID="64b4866351e93ad040bc618779f50695f0fb9bd8fea142a9ec2ef0aeaaba7d16" exitCode=0 Jul 23 12:10:55.391598 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:55.391252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624662448" event={"ID":"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f","Type":"ContainerDied","Data":"64b4866351e93ad040bc618779f50695f0fb9bd8fea142a9ec2ef0aeaaba7d16"} Jul 23 12:10:56.534932 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.534907 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:56.558798 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.558749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624662448_f86cc9f0-6ebd-4435-9dac-a923de7e1b0f/test-curl-7635-139826624662448/0.log" Jul 23 12:10:56.586104 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.586068 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624662448"] Jul 23 12:10:56.590821 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.590794 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624662448"] Jul 23 12:10:56.638374 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.638343 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtjxt\" (UniqueName: \"kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt\") pod \"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f\" (UID: \"f86cc9f0-6ebd-4435-9dac-a923de7e1b0f\") " Jul 23 12:10:56.640581 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.640560 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt" (OuterVolumeSpecName: "kube-api-access-qtjxt") pod "f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" (UID: "f86cc9f0-6ebd-4435-9dac-a923de7e1b0f"). InnerVolumeSpecName "kube-api-access-qtjxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:10:56.739545 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.739513 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjxt\" (UniqueName: \"kubernetes.io/projected/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f-kube-api-access-qtjxt\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:10:56.761101 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.761067 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:56.761505 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.761492 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" containerName="test-curl-7635-139826624662448" Jul 23 12:10:56.761546 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.761507 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" containerName="test-curl-7635-139826624662448" Jul 23 12:10:56.761585 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.761579 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" containerName="test-curl-7635-139826624662448" Jul 23 12:10:56.764632 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.764616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:56.772703 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.772678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:56.840094 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.840057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxj9\" (UniqueName: \"kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9\") pod \"test-curl-7635-139826624742720\" (UID: \"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:56.941375 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.941340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxj9\" (UniqueName: \"kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9\") pod \"test-curl-7635-139826624742720\" (UID: \"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:56.949670 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:56.949644 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxj9\" (UniqueName: \"kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9\") pod \"test-curl-7635-139826624742720\" (UID: \"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:57.076603 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:57.076510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:57.216095 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:57.216070 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:57.218268 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:10:57.218239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6b7276d_e8c6_4535_bb3a_f1f00c0049e3.slice/crio-ec9db371ef9e12aedea70bc3fbae5b424b88631f6b8fb50c108aa4d53d39584f WatchSource:0}: Error finding container ec9db371ef9e12aedea70bc3fbae5b424b88631f6b8fb50c108aa4d53d39584f: Status 404 returned error can't find the container with id ec9db371ef9e12aedea70bc3fbae5b424b88631f6b8fb50c108aa4d53d39584f Jul 23 12:10:57.405065 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:57.404966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624742720" event={"ID":"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3","Type":"ContainerStarted","Data":"ec9db371ef9e12aedea70bc3fbae5b424b88631f6b8fb50c108aa4d53d39584f"} Jul 23 12:10:57.406226 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:57.406210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:57.406296 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:57.406257 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0aa95708c36b3f1e3753ac1c13862234829690d5b828bc73f41a98bf630d509" Jul 23 12:10:57.416509 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:10:57.416478 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624662448\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:58.414459 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:58.414421 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" containerID="502c15be6514a06205408fe8f059b257e3570a2fcc226ed29fb37cc7490c3ee3" exitCode=0 Jul 23 12:10:58.414846 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:58.414464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624742720" event={"ID":"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3","Type":"ContainerDied","Data":"502c15be6514a06205408fe8f059b257e3570a2fcc226ed29fb37cc7490c3ee3"} Jul 23 12:10:58.432576 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:10:58.432534 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624662448\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" pod="opendatahub/test-curl-7635-139826624662448" Jul 23 12:10:58.543479 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:58.543440 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86cc9f0-6ebd-4435-9dac-a923de7e1b0f" path="/var/lib/kubelet/pods/f86cc9f0-6ebd-4435-9dac-a923de7e1b0f/volumes" Jul 23 12:10:59.581896 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.581872 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:59.601949 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.601921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624742720_c6b7276d-e8c6-4535-bb3a-f1f00c0049e3/test-curl-7635-139826624742720/0.log" Jul 23 12:10:59.625581 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.625550 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:59.629031 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.629007 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:59.666604 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.666577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdxj9\" (UniqueName: \"kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9\") pod \"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3\" (UID: \"c6b7276d-e8c6-4535-bb3a-f1f00c0049e3\") " Jul 23 12:10:59.668869 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.668839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9" (OuterVolumeSpecName: "kube-api-access-jdxj9") pod "c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" (UID: "c6b7276d-e8c6-4535-bb3a-f1f00c0049e3"). InnerVolumeSpecName "kube-api-access-jdxj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:10:59.751885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.751855 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:59.752313 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.752298 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" containerName="test-curl-7635-139826624742720" Jul 23 12:10:59.752374 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.752315 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" containerName="test-curl-7635-139826624742720" Jul 23 12:10:59.752409 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.752393 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" containerName="test-curl-7635-139826624742720" Jul 23 12:10:59.755402 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.755388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:59.762286 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.762140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:10:59.768296 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.768272 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxj9\" (UniqueName: \"kubernetes.io/projected/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3-kube-api-access-jdxj9\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:10:59.869553 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.869526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ps4\" (UniqueName: \"kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4\") pod \"test-curl-7635-139826624742720\" (UID: \"de4d770c-59d2-41f9-a652-f1482475f829\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:59.970213 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.970190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ps4\" (UniqueName: \"kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4\") pod \"test-curl-7635-139826624742720\" (UID: \"de4d770c-59d2-41f9-a652-f1482475f829\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:10:59.978400 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:10:59.978380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ps4\" (UniqueName: \"kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4\") pod \"test-curl-7635-139826624742720\" (UID: \"de4d770c-59d2-41f9-a652-f1482475f829\") " pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:00.066389 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.066364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:00.207064 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.207035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:11:00.428650 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.428617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624742720" event={"ID":"de4d770c-59d2-41f9-a652-f1482475f829","Type":"ContainerStarted","Data":"84878c540fd4884048081c4ff9c913f668199594193052144f79fc5887c132e3"} Jul 23 12:11:00.436867 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.436837 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9db371ef9e12aedea70bc3fbae5b424b88631f6b8fb50c108aa4d53d39584f" Jul 23 12:11:00.436867 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.436861 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:00.448542 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.448514 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7635-139826624742720" oldPodUID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" podUID="de4d770c-59d2-41f9-a652-f1482475f829" Jul 23 12:11:00.543926 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:00.543899 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b7276d-e8c6-4535-bb3a-f1f00c0049e3" path="/var/lib/kubelet/pods/c6b7276d-e8c6-4535-bb3a-f1f00c0049e3/volumes" Jul 23 12:11:01.444923 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:01.444887 2564 generic.go:356] "Generic (PLEG): container finished" podID="de4d770c-59d2-41f9-a652-f1482475f829" containerID="30b37262391a7deb664e6119bf3ffaf3338bd67c27ecbc26d032455c9997726e" exitCode=0 Jul 23 12:11:01.445335 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:01.444980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624742720" event={"ID":"de4d770c-59d2-41f9-a652-f1482475f829","Type":"ContainerDied","Data":"30b37262391a7deb664e6119bf3ffaf3338bd67c27ecbc26d032455c9997726e"} Jul 23 12:11:02.597563 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.597540 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:02.626043 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.626019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624742720_de4d770c-59d2-41f9-a652-f1482475f829/test-curl-7635-139826624742720/0.log" Jul 23 12:11:02.657393 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.657362 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:11:02.662465 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.662440 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624742720"] Jul 23 12:11:02.693866 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.693838 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ps4\" (UniqueName: \"kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4\") pod \"de4d770c-59d2-41f9-a652-f1482475f829\" (UID: \"de4d770c-59d2-41f9-a652-f1482475f829\") " Jul 23 12:11:02.696061 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.696037 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4" (OuterVolumeSpecName: "kube-api-access-b2ps4") pod "de4d770c-59d2-41f9-a652-f1482475f829" (UID: "de4d770c-59d2-41f9-a652-f1482475f829"). InnerVolumeSpecName "kube-api-access-b2ps4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:02.791518 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.791483 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:02.791966 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.791951 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4d770c-59d2-41f9-a652-f1482475f829" containerName="test-curl-7635-139826624742720" Jul 23 12:11:02.792016 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.791967 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4d770c-59d2-41f9-a652-f1482475f829" containerName="test-curl-7635-139826624742720" Jul 23 12:11:02.792057 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.792044 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4d770c-59d2-41f9-a652-f1482475f829" containerName="test-curl-7635-139826624742720" Jul 23 12:11:02.795199 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.795182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:02.795306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.795265 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ps4\" (UniqueName: \"kubernetes.io/projected/de4d770c-59d2-41f9-a652-f1482475f829-kube-api-access-b2ps4\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:02.802479 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.802449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:02.895870 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.895833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gk9\" (UniqueName: \"kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9\") pod \"test-curl-7635-139826624743728\" (UID: \"c6dfce34-dff1-4f40-b04e-f10b67eccd2e\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:02.996809 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:02.996775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gk9\" (UniqueName: \"kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9\") pod \"test-curl-7635-139826624743728\" (UID: \"c6dfce34-dff1-4f40-b04e-f10b67eccd2e\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:03.005976 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.005940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gk9\" (UniqueName: \"kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9\") pod \"test-curl-7635-139826624743728\" (UID: \"c6dfce34-dff1-4f40-b04e-f10b67eccd2e\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:03.106151 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.106119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:03.244882 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.244856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:03.246533 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:11:03.246508 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6dfce34_dff1_4f40_b04e_f10b67eccd2e.slice/crio-f40689b4da9f8a0be98fa33ab0d631a0adf172d082a69812ac3c13edaea45956 WatchSource:0}: Error finding container f40689b4da9f8a0be98fa33ab0d631a0adf172d082a69812ac3c13edaea45956: Status 404 returned error can't find the container with id f40689b4da9f8a0be98fa33ab0d631a0adf172d082a69812ac3c13edaea45956 Jul 23 12:11:03.459557 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.459529 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84878c540fd4884048081c4ff9c913f668199594193052144f79fc5887c132e3" Jul 23 12:11:03.459557 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.459533 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:03.460901 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:03.460868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624743728" event={"ID":"c6dfce34-dff1-4f40-b04e-f10b67eccd2e","Type":"ContainerStarted","Data":"f40689b4da9f8a0be98fa33ab0d631a0adf172d082a69812ac3c13edaea45956"} Jul 23 12:11:03.481013 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:03.480975 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624742720\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="de4d770c-59d2-41f9-a652-f1482475f829" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:04.468607 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:04.468571 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" containerID="a1778416979b42b6fa1d7d61e6e4f0a5a2064ef184bbd4d78517b171ab20e30f" exitCode=0 Jul 23 12:11:04.469028 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:04.468643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624743728" event={"ID":"c6dfce34-dff1-4f40-b04e-f10b67eccd2e","Type":"ContainerDied","Data":"a1778416979b42b6fa1d7d61e6e4f0a5a2064ef184bbd4d78517b171ab20e30f"} Jul 23 12:11:04.485121 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:04.485082 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624742720\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="de4d770c-59d2-41f9-a652-f1482475f829" pod="opendatahub/test-curl-7635-139826624742720" Jul 23 12:11:04.543142 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:04.543109 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4d770c-59d2-41f9-a652-f1482475f829" path="/var/lib/kubelet/pods/de4d770c-59d2-41f9-a652-f1482475f829/volumes" Jul 23 12:11:05.615265 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.615241 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:05.635301 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.635277 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624743728_c6dfce34-dff1-4f40-b04e-f10b67eccd2e/test-curl-7635-139826624743728/0.log" Jul 23 12:11:05.659394 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.659361 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:05.662701 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.662678 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:05.722936 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.722899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7gk9\" (UniqueName: \"kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9\") pod \"c6dfce34-dff1-4f40-b04e-f10b67eccd2e\" (UID: \"c6dfce34-dff1-4f40-b04e-f10b67eccd2e\") " Jul 23 12:11:05.725155 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.725131 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9" (OuterVolumeSpecName: "kube-api-access-b7gk9") pod "c6dfce34-dff1-4f40-b04e-f10b67eccd2e" (UID: "c6dfce34-dff1-4f40-b04e-f10b67eccd2e"). InnerVolumeSpecName "kube-api-access-b7gk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:05.802922 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.802891 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:05.803345 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.803332 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" containerName="test-curl-7635-139826624743728" Jul 23 12:11:05.803401 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.803347 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" containerName="test-curl-7635-139826624743728" Jul 23 12:11:05.803441 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.803422 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" containerName="test-curl-7635-139826624743728" Jul 23 12:11:05.806668 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.806642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:05.813705 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.813680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:05.824281 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.824258 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gk9\" (UniqueName: \"kubernetes.io/projected/c6dfce34-dff1-4f40-b04e-f10b67eccd2e-kube-api-access-b7gk9\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:05.925115 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:05.925073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb7n\" (UniqueName: \"kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n\") pod \"test-curl-7635-139826624743728\" (UID: \"ca972f29-3cd0-424d-94cd-63c64b3e043a\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:06.025904 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.025812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb7n\" (UniqueName: \"kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n\") pod \"test-curl-7635-139826624743728\" (UID: \"ca972f29-3cd0-424d-94cd-63c64b3e043a\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:06.034802 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.034752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb7n\" (UniqueName: \"kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n\") pod \"test-curl-7635-139826624743728\" (UID: \"ca972f29-3cd0-424d-94cd-63c64b3e043a\") " pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:06.118446 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.118412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:06.263639 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.263610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:06.265064 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:11:06.265043 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca972f29_3cd0_424d_94cd_63c64b3e043a.slice/crio-2c7a003563a6956a5db90a34c64d0043cd07fd8fa2c9ce4e30b1512cbdb9b028 WatchSource:0}: Error finding container 2c7a003563a6956a5db90a34c64d0043cd07fd8fa2c9ce4e30b1512cbdb9b028: Status 404 returned error can't find the container with id 2c7a003563a6956a5db90a34c64d0043cd07fd8fa2c9ce4e30b1512cbdb9b028 Jul 23 12:11:06.483573 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.483534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624743728" event={"ID":"ca972f29-3cd0-424d-94cd-63c64b3e043a","Type":"ContainerStarted","Data":"2c7a003563a6956a5db90a34c64d0043cd07fd8fa2c9ce4e30b1512cbdb9b028"} Jul 23 12:11:06.484943 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.484923 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40689b4da9f8a0be98fa33ab0d631a0adf172d082a69812ac3c13edaea45956" Jul 23 12:11:06.484943 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.484932 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:06.496616 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.496588 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7635-139826624743728" oldPodUID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" Jul 23 12:11:06.544161 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:06.544137 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6dfce34-dff1-4f40-b04e-f10b67eccd2e" path="/var/lib/kubelet/pods/c6dfce34-dff1-4f40-b04e-f10b67eccd2e/volumes" Jul 23 12:11:07.492349 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:07.492314 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca972f29-3cd0-424d-94cd-63c64b3e043a" containerID="9b449f719e5971855f16b8b7bed395879427d2b18b54ac3140dc6aced041ed82" exitCode=0 Jul 23 12:11:07.492779 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:07.492385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-139826624743728" event={"ID":"ca972f29-3cd0-424d-94cd-63c64b3e043a","Type":"ContainerDied","Data":"9b449f719e5971855f16b8b7bed395879427d2b18b54ac3140dc6aced041ed82"} Jul 23 12:11:08.646595 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.646567 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:08.668822 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.668795 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7635-139826624743728_ca972f29-3cd0-424d-94cd-63c64b3e043a/test-curl-7635-139826624743728/0.log" Jul 23 12:11:08.696225 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.696201 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:08.699389 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.699369 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-139826624743728"] Jul 23 12:11:08.754163 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.754141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwb7n\" (UniqueName: \"kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n\") pod \"ca972f29-3cd0-424d-94cd-63c64b3e043a\" (UID: \"ca972f29-3cd0-424d-94cd-63c64b3e043a\") " Jul 23 12:11:08.756313 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.756255 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n" (OuterVolumeSpecName: "kube-api-access-dwb7n") pod "ca972f29-3cd0-424d-94cd-63c64b3e043a" (UID: "ca972f29-3cd0-424d-94cd-63c64b3e043a"). InnerVolumeSpecName "kube-api-access-dwb7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:08.855780 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:08.855736 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwb7n\" (UniqueName: \"kubernetes.io/projected/ca972f29-3cd0-424d-94cd-63c64b3e043a-kube-api-access-dwb7n\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:09.432086 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:09.432056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:09.512450 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:09.512423 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7a003563a6956a5db90a34c64d0043cd07fd8fa2c9ce4e30b1512cbdb9b028" Jul 23 12:11:09.512450 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:09.512429 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:09.522309 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:09.522282 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-139826624743728\" is forbidden: User \"system:node:ip-10-0-128-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-30.ec2.internal' and this object" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" pod="opendatahub/test-curl-7635-139826624743728" Jul 23 12:11:09.534853 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:09.534832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:10.543186 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:10.543155 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" path="/var/lib/kubelet/pods/ca972f29-3cd0-424d-94cd-63c64b3e043a/volumes" Jul 23 12:11:19.434671 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:19.434637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:23.995890 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:23.995858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:24.409013 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:24.408925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:30.590181 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:30.590147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:30.695649 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:30.695610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:30.860945 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:30.860855 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:11:30.861133 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:30.861106 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" containerID="cri-o://71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac" gracePeriod=30 Jul 23 12:11:30.890408 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:30.890375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:31.125806 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:31.125700 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:31.845668 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:31.845631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:31.942796 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:31.942743 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:36.121183 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.121162 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:11:36.219812 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.219717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.219812 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.219800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmrxr\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220019 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.219871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220019 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220000 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220135 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220135 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220093 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220135 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220128 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220183 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data\") pod \"512b52fa-5713-497c-804e-4afdefadbd04\" (UID: \"512b52fa-5713-497c-804e-4afdefadbd04\") " Jul 23 12:11:36.220284 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:36.220427 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:11:36.220800 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220725 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:36.220952 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220835 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.220952 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220860 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512b52fa-5713-497c-804e-4afdefadbd04-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.221066 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.220957 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:36.221197 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.221167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data" (OuterVolumeSpecName: "istio-data") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:36.222120 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.222098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr" (OuterVolumeSpecName: "kube-api-access-vmrxr") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "kube-api-access-vmrxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:36.222326 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.222306 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:36.222536 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.222517 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:11:36.222579 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.222541 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token" (OuterVolumeSpecName: "istio-token") pod "512b52fa-5713-497c-804e-4afdefadbd04" (UID: "512b52fa-5713-497c-804e-4afdefadbd04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:36.322117 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322084 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322117 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322113 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322126 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322136 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512b52fa-5713-497c-804e-4afdefadbd04-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322147 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrxr\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-kube-api-access-vmrxr\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322160 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512b52fa-5713-497c-804e-4afdefadbd04-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.322306 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.322172 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512b52fa-5713-497c-804e-4afdefadbd04-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:36.687876 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.687845 2564 generic.go:356] "Generic (PLEG): container finished" podID="512b52fa-5713-497c-804e-4afdefadbd04" containerID="71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac" exitCode=0 Jul 23 12:11:36.688024 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.687910 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" Jul 23 12:11:36.688024 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.687911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerDied","Data":"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac"} Jul 23 12:11:36.688114 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.688046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv" event={"ID":"512b52fa-5713-497c-804e-4afdefadbd04","Type":"ContainerDied","Data":"d5fcd8ae41825b75a73d212421d94fc4fc40589b477244039f08b5c9b5fabefc"} Jul 23 12:11:36.688114 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.688083 2564 scope.go:117] "RemoveContainer" containerID="71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac" Jul 23 12:11:36.697729 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.697712 2564 scope.go:117] "RemoveContainer" containerID="d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0" Jul 23 12:11:36.706146 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.706131 2564 scope.go:117] "RemoveContainer" containerID="71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac" Jul 23 12:11:36.706397 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:36.706378 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac\": container with ID starting with 71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac not found: ID does not exist" containerID="71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac" Jul 23 12:11:36.706460 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.706404 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac"} err="failed to get container status \"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac\": rpc error: code = NotFound desc = could not find container \"71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac\": container with ID starting with 71894cd2b6d39c73f0d9e3d4f17783c62b9a85820d5b48f0bc6a9ed7e19fcdac not found: ID does not exist" Jul 23 12:11:36.706460 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.706419 2564 scope.go:117] "RemoveContainer" containerID="d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0" Jul 23 12:11:36.706668 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:36.706651 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0\": container with ID starting with d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0 not found: ID does not exist" containerID="d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0" Jul 23 12:11:36.706711 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.706673 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0"} err="failed to get container status \"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0\": rpc error: code = NotFound desc = could not find container \"d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0\": container with ID starting with d395d1436f80d64b03731c0e096489989500452257640036fe9b6d8914d061b0 not found: ID does not exist" Jul 23 12:11:36.711685 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.711663 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:11:36.714923 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:36.714901 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7cc2e5-openshift-default-896bf6795-wlpmv"] Jul 23 12:11:38.543098 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:38.543060 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512b52fa-5713-497c-804e-4afdefadbd04" path="/var/lib/kubelet/pods/512b52fa-5713-497c-804e-4afdefadbd04/volumes" Jul 23 12:11:41.761167 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:41.761131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:46.327223 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:46.327185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:46.736957 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:46.736921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:46.759414 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:46.759387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:11:46.764201 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:46.764178 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4kjj5_7ecbc862-e2ba-430c-9bab-dfcb931bc27b/ovn-acl-logging/0.log" Jul 23 12:11:52.833377 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:52.833337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:52.941885 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:52.941853 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:53.077468 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:53.077434 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:11:53.077662 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:53.077634 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" containerID="cri-o://4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4" gracePeriod=30 Jul 23 12:11:53.096495 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:53.096416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:53.329447 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:53.329418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:11:53.757158 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:53.757126 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d8df7d8c5-dtj2h_31ae84b2-8fdb-40c2-b739-e2651c7e1815/authorino/0.log" Jul 23 12:11:58.332814 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.332786 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:11:58.430924 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.430821 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.430924 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.430902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.430928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.430981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431028 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431173 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gwm9\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431473 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token\") pod \"b65b8456-66f3-4cf5-886f-a80d05f8b766\" (UID: \"b65b8456-66f3-4cf5-886f-a80d05f8b766\") " Jul 23 12:11:58.431473 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431248 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:11:58.431473 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431435 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:58.431473 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:58.431473 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431343 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:58.431727 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431552 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-certs\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.431727 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431580 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b8456-66f3-4cf5-886f-a80d05f8b766-istiod-ca-cert\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.431727 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431603 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-workload-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.431727 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431611 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-credential-socket\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.431935 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.431837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data" (OuterVolumeSpecName: "istio-data") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:58.433545 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.433522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token" (OuterVolumeSpecName: "istio-token") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:58.433628 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.433568 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9" (OuterVolumeSpecName: "kube-api-access-8gwm9") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "kube-api-access-8gwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:58.433821 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.433805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:58.433821 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.433810 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b65b8456-66f3-4cf5-886f-a80d05f8b766" (UID: "b65b8456-66f3-4cf5-886f-a80d05f8b766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:11:58.532691 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.532657 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-envoy\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.532691 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.532684 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-podinfo\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.532691 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.532695 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwm9\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-kube-api-access-8gwm9\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.532963 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.532704 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-token\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.532963 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.532712 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b8456-66f3-4cf5-886f-a80d05f8b766-istio-data\") on node \"ip-10-0-128-30.ec2.internal\" DevicePath \"\"" Jul 23 12:11:58.828880 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.828842 2564 generic.go:356] "Generic (PLEG): container finished" podID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerID="4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4" exitCode=0 Jul 23 12:11:58.829081 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.828901 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" Jul 23 12:11:58.829081 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.828930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerDied","Data":"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4"} Jul 23 12:11:58.829081 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.828985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25" event={"ID":"b65b8456-66f3-4cf5-886f-a80d05f8b766","Type":"ContainerDied","Data":"60fe96093177926fcac9680298e4266339b5286841baa0989fc09af46ea230a1"} Jul 23 12:11:58.829081 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.829007 2564 scope.go:117] "RemoveContainer" containerID="4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4" Jul 23 12:11:58.838563 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.838537 2564 scope.go:117] "RemoveContainer" containerID="d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a" Jul 23 12:11:58.847397 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.847378 2564 scope.go:117] "RemoveContainer" containerID="4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4" Jul 23 12:11:58.847708 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:58.847689 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4\": container with ID starting with 4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4 not found: ID does not exist" containerID="4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4" Jul 23 12:11:58.847789 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.847718 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4"} err="failed to get container status \"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4\": rpc error: code = NotFound desc = could not find container \"4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4\": container with ID starting with 4a9b30ad37ca51389789bef266997415aee0623c2efe5cd153e4a5f875200cd4 not found: ID does not exist" Jul 23 12:11:58.847789 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.847735 2564 scope.go:117] "RemoveContainer" containerID="d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a" Jul 23 12:11:58.848000 ip-10-0-128-30 kubenswrapper[2564]: E0723 12:11:58.847982 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a\": container with ID starting with d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a not found: ID does not exist" containerID="d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a" Jul 23 12:11:58.848058 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.848010 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a"} err="failed to get container status \"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a\": rpc error: code = NotFound desc = could not find container \"d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a\": container with ID starting with d88c0d99adb56a54fe94cfa9e301d2ff610a6e2e9207a1cb5ec83f13d138389a not found: ID does not exist" Jul 23 12:11:58.850206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.850186 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:11:58.853708 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:58.853687 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-77f87c-openshift-default-59c594cdb7-b6t25"] Jul 23 12:11:59.266830 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.266786 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mlxzc_39e6c624-4b32-4437-852b-561aaaf22c21/manager/0.log" Jul 23 12:11:59.395955 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.395925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8pnpg_6f5ebd27-aa6d-43e9-8c2f-914ce64afa55/manager/0.log" Jul 23 12:11:59.516229 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.516197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79c669ccc6-md9jl_a77a06d8-fd77-4f51-b333-abdd663c60d4/manager/0.log" Jul 23 12:11:59.634002 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.633927 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mvmhl_80fb3559-341d-4ea0-8596-4a6827ba824a/server/0.log" Jul 23 12:11:59.753652 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.753623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mm2zj_fd486d6b-29c0-4374-a224-d4e83b2c4495/manager/0.log" Jul 23 12:11:59.865877 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.865850 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-slnv4_f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7/copy-manifests/0.log" Jul 23 12:11:59.872005 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:11:59.871986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-slnv4_f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7/manager/0.log" Jul 23 12:12:00.543932 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:00.543900 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" path="/var/lib/kubelet/pods/b65b8456-66f3-4cf5-886f-a80d05f8b766/volumes" Jul 23 12:12:00.637031 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:00.636999 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746785-cdh2p_1677f2bc-bf2d-44ed-a536-05ef54238942/cleanup/0.log" Jul 23 12:12:00.751071 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:00.751041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746800-lxj84_ada63923-294a-4b2e-9bab-fd32f6e22013/cleanup/0.log" Jul 23 12:12:00.862667 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:00.862589 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ws8rz_54288d3e-132d-4e08-b035-75091ff814ad/postgres/0.log" Jul 23 12:12:01.683373 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.683348 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/util/0.log" Jul 23 12:12:01.691094 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.691072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/pull/0.log" Jul 23 12:12:01.698527 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.698507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/extract/0.log" Jul 23 12:12:01.813307 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.813280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/util/0.log" Jul 23 12:12:01.820478 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.820457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/pull/0.log" Jul 23 12:12:01.826875 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.826857 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/extract/0.log" Jul 23 12:12:01.949454 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.949384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/util/0.log" Jul 23 12:12:01.955978 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.955956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/pull/0.log" Jul 23 12:12:01.962574 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:01.962555 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/extract/0.log" Jul 23 12:12:02.078604 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.078573 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/pull/0.log" Jul 23 12:12:02.086185 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.086167 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/extract/0.log" Jul 23 12:12:02.092726 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.092705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/util/0.log" Jul 23 12:12:02.213359 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.213275 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d8df7d8c5-dtj2h_31ae84b2-8fdb-40c2-b739-e2651c7e1815/authorino/0.log" Jul 23 12:12:02.327731 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.327703 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fkvmq_d00cd354-8afa-434c-b778-64d0475268d1/manager/0.log" Jul 23 12:12:02.449224 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.449194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p5g8k_b8a63fe9-3653-4352-8268-fda4bf5f2d68/manager/0.log" Jul 23 12:12:02.558338 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.558310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9sqzk_9a3b996e-4708-4a70-ba2f-f5567cd666a8/kuadrant-console-plugin/0.log" Jul 23 12:12:02.686720 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.682007 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-84sh9_387dac0e-9014-4648-b8f0-06c3fafbaa9a/registry-server/0.log" Jul 23 12:12:02.808300 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:02.808263 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-642dr_7c4abd77-94f1-479a-8a73-36e1b5d3f59a/manager/0.log" Jul 23 12:12:03.041559 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:03.041533 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cbx5k_a606d3b3-4b05-4ec9-ba75-eefd0736f411/manager/0.log" Jul 23 12:12:03.759678 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:03.759648 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-ph858_260ad48d-5e1c-47b5-a8fe-a0cf6c233712/kube-auth-proxy/0.log" Jul 23 12:12:04.005428 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:04.005396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xzrb4_4ecc94e6-3546-4d27-85dc-973a6742c957/payload-pre-processing/0.log" Jul 23 12:12:04.130799 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:04.130704 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-b8z4d_70b34262-a94d-432d-a805-8178c5fa049e/payload-processing/0.log" Jul 23 12:12:04.745955 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:04.745931 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qglqf_6c425ab8-0da3-4029-8406-1667e9262aeb/main/0.log" Jul 23 12:12:10.881710 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.881674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xw2jd/must-gather-tnv2k"] Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882083 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" containerName="test-curl-7635-139826624743728" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882095 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" containerName="test-curl-7635-139826624743728" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882105 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882110 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882122 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882127 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882143 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882149 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882206 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882206 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882216 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882223 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca972f29-3cd0-424d-94cd-63c64b3e043a" containerName="test-curl-7635-139826624743728" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882229 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882234 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65b8456-66f3-4cf5-886f-a80d05f8b766" containerName="istio-proxy" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882293 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.882494 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.882300 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="512b52fa-5713-497c-804e-4afdefadbd04" containerName="istio-proxy" Jul 23 12:12:10.885216 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.885197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:10.888002 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.887980 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xw2jd\"/\"kube-root-ca.crt\"" Jul 23 12:12:10.888896 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.888878 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xw2jd\"/\"default-dockercfg-k7f76\"" Jul 23 12:12:10.889004 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.888964 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xw2jd\"/\"openshift-service-ca.crt\"" Jul 23 12:12:10.897462 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.897440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/must-gather-tnv2k"] Jul 23 12:12:10.900255 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.900235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:12:10.931026 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.930989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba7555e1-1506-437a-a351-9c06a3b80874-must-gather-output\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:10.931177 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.931120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx59\" (UniqueName: \"kubernetes.io/projected/ba7555e1-1506-437a-a351-9c06a3b80874-kube-api-access-qxx59\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:10.966409 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:10.966368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:12:11.032166 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.032135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx59\" (UniqueName: \"kubernetes.io/projected/ba7555e1-1506-437a-a351-9c06a3b80874-kube-api-access-qxx59\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:11.032358 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.032195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba7555e1-1506-437a-a351-9c06a3b80874-must-gather-output\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:11.032504 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.032485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba7555e1-1506-437a-a351-9c06a3b80874-must-gather-output\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:11.041415 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.041393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx59\" (UniqueName: \"kubernetes.io/projected/ba7555e1-1506-437a-a351-9c06a3b80874-kube-api-access-qxx59\") pod \"must-gather-tnv2k\" (UID: \"ba7555e1-1506-437a-a351-9c06a3b80874\") " pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:11.195610 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.195512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" Jul 23 12:12:11.332405 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.332365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/must-gather-tnv2k"] Jul 23 12:12:11.333616 ip-10-0-128-30 kubenswrapper[2564]: W0723 12:12:11.333583 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7555e1_1506_437a_a351_9c06a3b80874.slice/crio-60dbf8b12dec7bf2e1a0fbc0e64224e8b8a6d53bd65085b04b29e74cc7da59ed WatchSource:0}: Error finding container 60dbf8b12dec7bf2e1a0fbc0e64224e8b8a6d53bd65085b04b29e74cc7da59ed: Status 404 returned error can't find the container with id 60dbf8b12dec7bf2e1a0fbc0e64224e8b8a6d53bd65085b04b29e74cc7da59ed Jul 23 12:12:11.914238 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:11.914202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" event={"ID":"ba7555e1-1506-437a-a351-9c06a3b80874","Type":"ContainerStarted","Data":"60dbf8b12dec7bf2e1a0fbc0e64224e8b8a6d53bd65085b04b29e74cc7da59ed"} Jul 23 12:12:12.922558 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:12.922521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" event={"ID":"ba7555e1-1506-437a-a351-9c06a3b80874","Type":"ContainerStarted","Data":"ad47a6c49fb967d4cd9ef9457ab9292ac155d30803839b6384f1a1473d14c165"} Jul 23 12:12:12.923017 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:12.922572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" event={"ID":"ba7555e1-1506-437a-a351-9c06a3b80874","Type":"ContainerStarted","Data":"194dee56039418f8fe38366b0d570fea2a7f8e7c11441d5b0d68eee0c6572bc9"} Jul 23 12:12:12.940440 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:12.940372 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xw2jd/must-gather-tnv2k" podStartSLOduration=2.197542797 podStartE2EDuration="2.940347004s" totalImagesPullingTime="742.804207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:12.938781551 +0000 UTC m=+2726.920779057" watchObservedRunningTime="2026-07-23 12:12:12.940347004 +0000 UTC m=+2726.922344510" Jul 23 12:12:13.718401 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:13.718375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hfvsb_ab9726ae-8666-45a9-89d2-91e32363b1ee/global-pull-secret-syncer/0.log" Jul 23 12:12:13.833047 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:13.833011 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" Jul 23 12:12:13.921524 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:13.921464 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-30.ec2.internal_e524dd64ccddd7fadb096176a3752af3/haproxy/0.log" Jul 23 12:12:18.151319 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.151293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/extract/0.log" Jul 23 12:12:18.175400 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.175291 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/util/0.log" Jul 23 12:12:18.200143 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.200107 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ckk9s_030b492e-4ffb-4771-8c04-13d2c8568bee/pull/0.log" Jul 23 12:12:18.242467 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.242433 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/extract/0.log" Jul 23 12:12:18.266009 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.265982 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/util/0.log" Jul 23 12:12:18.289349 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.289292 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gmqch_bc683b81-d698-40bc-9405-1a315e533879/pull/0.log" Jul 23 12:12:18.333497 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.333471 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/extract/0.log" Jul 23 12:12:18.359430 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.359406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/util/0.log" Jul 23 12:12:18.382906 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.382881 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738ggtw_c66bcd69-3978-4be5-b0d2-010f16e3bf78/pull/0.log" Jul 23 12:12:18.413581 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.413497 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/extract/0.log" Jul 23 12:12:18.443449 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.443424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/util/0.log" Jul 23 12:12:18.467983 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.467954 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12drjg_ef7e760e-6bf3-40d7-ab84-3bf455475ffc/pull/0.log" Jul 23 12:12:18.956831 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.956800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d8df7d8c5-dtj2h_31ae84b2-8fdb-40c2-b739-e2651c7e1815/authorino/0.log" Jul 23 12:12:18.989594 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:18.989567 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fkvmq_d00cd354-8afa-434c-b778-64d0475268d1/manager/0.log" Jul 23 12:12:19.025855 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:19.025820 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p5g8k_b8a63fe9-3653-4352-8268-fda4bf5f2d68/manager/0.log" Jul 23 12:12:19.058074 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:19.058045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9sqzk_9a3b996e-4708-4a70-ba2f-f5567cd666a8/kuadrant-console-plugin/0.log" Jul 23 12:12:19.107855 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:19.107818 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-84sh9_387dac0e-9014-4648-b8f0-06c3fafbaa9a/registry-server/0.log" Jul 23 12:12:19.287873 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:19.287720 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-642dr_7c4abd77-94f1-479a-8a73-36e1b5d3f59a/manager/0.log" Jul 23 12:12:19.403901 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:19.403346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cbx5k_a606d3b3-4b05-4ec9-ba75-eefd0736f411/manager/0.log" Jul 23 12:12:21.196220 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.196193 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/alertmanager/0.log" Jul 23 12:12:21.238905 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.238879 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/config-reloader/0.log" Jul 23 12:12:21.300964 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.300852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/kube-rbac-proxy-web/0.log" Jul 23 12:12:21.339786 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.333481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/kube-rbac-proxy/0.log" Jul 23 12:12:21.369385 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.369362 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/kube-rbac-proxy-metric/0.log" Jul 23 12:12:21.396929 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.396898 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/prom-label-proxy/0.log" Jul 23 12:12:21.425857 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.425711 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f4b88c7-03ba-409d-a796-ed7d85070461/init-config-reloader/0.log" Jul 23 12:12:21.506946 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.506896 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mf675_e10fae66-1c9b-4bd3-9c54-8259a0e6defd/kube-state-metrics/0.log" Jul 23 12:12:21.527798 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.527741 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mf675_e10fae66-1c9b-4bd3-9c54-8259a0e6defd/kube-rbac-proxy-main/0.log" Jul 23 12:12:21.558054 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.558026 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mf675_e10fae66-1c9b-4bd3-9c54-8259a0e6defd/kube-rbac-proxy-self/0.log" Jul 23 12:12:21.873786 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.871390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fsl4_89e743ad-372a-4439-b672-575b7bfc6f32/node-exporter/0.log" Jul 23 12:12:21.890022 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.889997 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fsl4_89e743ad-372a-4439-b672-575b7bfc6f32/kube-rbac-proxy/0.log" Jul 23 12:12:21.919439 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:21.919409 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fsl4_89e743ad-372a-4439-b672-575b7bfc6f32/init-textfile/0.log" Jul 23 12:12:22.433988 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.433961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/thanos-query/0.log" Jul 23 12:12:22.457965 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.457930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/kube-rbac-proxy-web/0.log" Jul 23 12:12:22.491055 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.490971 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/kube-rbac-proxy/0.log" Jul 23 12:12:22.514296 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.514241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/prom-label-proxy/0.log" Jul 23 12:12:22.545534 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.545498 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/kube-rbac-proxy-rules/0.log" Jul 23 12:12:22.576366 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:22.576331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf88f8b97-ks9jc_1a0f6faa-b0a6-4a57-834b-2b55aaae402f/kube-rbac-proxy-metrics/0.log" Jul 23 12:12:23.844058 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.844008 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl"] Jul 23 12:12:23.853449 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.853420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:23.869673 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.869641 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl"] Jul 23 12:12:23.999064 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.999021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-proc\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:23.999263 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.999092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-podres\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:23.999263 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.999130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbr2m\" (UniqueName: \"kubernetes.io/projected/024782ca-d941-4d8a-9112-5147b5f607e6-kube-api-access-kbr2m\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:23.999263 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.999192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-lib-modules\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:23.999263 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:23.999214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-sys\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100269 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-proc\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100269 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-podres\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbr2m\" (UniqueName: \"kubernetes.io/projected/024782ca-d941-4d8a-9112-5147b5f607e6-kube-api-access-kbr2m\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-lib-modules\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-sys\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-proc\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100599 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-sys\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100889 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-lib-modules\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.100889 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.100696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/024782ca-d941-4d8a-9112-5147b5f607e6-podres\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.110422 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.110393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbr2m\" (UniqueName: \"kubernetes.io/projected/024782ca-d941-4d8a-9112-5147b5f607e6-kube-api-access-kbr2m\") pod \"perf-node-gather-daemonset-2kxdl\" (UID: \"024782ca-d941-4d8a-9112-5147b5f607e6\") " pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.178617 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.178581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:24.374615 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:24.374514 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl"] Jul 23 12:12:25.043787 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:25.043039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" event={"ID":"024782ca-d941-4d8a-9112-5147b5f607e6","Type":"ContainerStarted","Data":"cb5bb69a73a63ee9c5c568162348f2758a48f12e3743525a0a521573c5597244"} Jul 23 12:12:25.044274 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:25.043808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" event={"ID":"024782ca-d941-4d8a-9112-5147b5f607e6","Type":"ContainerStarted","Data":"309e6618b1b592761924343430e4d6707c987fb7dfe1da5f29f0c604ae09185a"} Jul 23 12:12:25.044274 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:25.043848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:25.066386 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:25.066333 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" podStartSLOduration=2.066316842 podStartE2EDuration="2.066316842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:25.064651881 +0000 UTC m=+2739.046649384" watchObservedRunningTime="2026-07-23 12:12:25.066316842 +0000 UTC m=+2739.048314344" Jul 23 12:12:25.101688 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:25.101635 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74d8d5c76b-cq2z2_c9761335-16b5-4a27-8f71-1d18e0c7d871/console/0.log" Jul 23 12:12:26.936636 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:26.936598 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4xtr_507d5574-83f2-439f-9c08-bbcb51b1be6f/dns/0.log" Jul 23 12:12:26.983096 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:26.983069 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4xtr_507d5574-83f2-439f-9c08-bbcb51b1be6f/kube-rbac-proxy/0.log" Jul 23 12:12:27.105183 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:27.105135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6xkh9_7655ae45-899d-45ed-a10f-6286660f2c51/dns-node-resolver/0.log" Jul 23 12:12:27.724527 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:27.724501 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dd4b8fc4b-75vcl_8436097c-ebd7-4aa1-b7f0-c4cc7b056a5b/registry/0.log" Jul 23 12:12:27.767693 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:27.767653 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs9l8_04cb6eb4-10c9-4b3c-9aaf-4c6ce51ccc16/node-ca/0.log" Jul 23 12:12:29.029603 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.029577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-ph858_260ad48d-5e1c-47b5-a8fe-a0cf6c233712/kube-auth-proxy/0.log" Jul 23 12:12:29.127316 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.127288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xzrb4_4ecc94e6-3546-4d27-85dc-973a6742c957/payload-pre-processing/0.log" Jul 23 12:12:29.161152 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.161117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-b8z4d_70b34262-a94d-432d-a805-8178c5fa049e/payload-processing/0.log" Jul 23 12:12:29.562740 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.562705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:12:29.626271 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.626231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:12:29.999829 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:29.999797 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rmglm_a6d788b9-c6ed-46fd-81f9-89ac7de3a1dd/serve-healthcheck-canary/0.log" Jul 23 12:12:30.632961 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:30.632932 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdztf_6f5bc939-1678-41f0-9073-3a82fbaa94ed/kube-rbac-proxy/0.log" Jul 23 12:12:30.660899 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:30.660850 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdztf_6f5bc939-1678-41f0-9073-3a82fbaa94ed/exporter/0.log" Jul 23 12:12:30.690570 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:30.690539 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdztf_6f5bc939-1678-41f0-9073-3a82fbaa94ed/extractor/0.log" Jul 23 12:12:31.067917 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:31.067889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xw2jd/perf-node-gather-daemonset-2kxdl" Jul 23 12:12:31.836500 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:31.836466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96klp_01887123-dc9b-428d-967f-8d195ad04afe/konnectivity-agent/0.log" limit=1024 Jul 23 12:12:33.280553 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.280522 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mlxzc_39e6c624-4b32-4437-852b-561aaaf22c21/manager/0.log" Jul 23 12:12:33.548799 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.548686 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8pnpg_6f5ebd27-aa6d-43e9-8c2f-914ce64afa55/manager/0.log" Jul 23 12:12:33.708459 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.707630 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79c669ccc6-md9jl_a77a06d8-fd77-4f51-b333-abdd663c60d4/manager/0.log" Jul 23 12:12:33.738526 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.738500 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mvmhl_80fb3559-341d-4ea0-8596-4a6827ba824a/server/0.log" Jul 23 12:12:33.845450 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.845359 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mm2zj_fd486d6b-29c0-4374-a224-d4e83b2c4495/manager/0.log" Jul 23 12:12:33.875258 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.875168 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-slnv4_f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7/manager/0.log" Jul 23 12:12:33.902018 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:33.901965 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-674bfcc66c-slnv4_f64dcc09-3f9a-4f11-b85e-3554a9a5cdd7/copy-manifests/0.log" Jul 23 12:12:34.543781 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:34.543723 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-ph858_260ad48d-5e1c-47b5-a8fe-a0cf6c233712/kube-auth-proxy/0.log" Jul 23 12:12:34.625609 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:34.625550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xzrb4_4ecc94e6-3546-4d27-85dc-973a6742c957/payload-pre-processing/0.log" Jul 23 12:12:34.648420 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:34.648384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-b8z4d_70b34262-a94d-432d-a805-8178c5fa049e/payload-processing/0.log" Jul 23 12:12:35.382321 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:35.382239 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s4hpr_70996867-f248-4731-a661-482f6cef0105/openshift-lws-operator/0.log" Jul 23 12:12:36.039878 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:36.039853 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766d5cd5c5-ph858_260ad48d-5e1c-47b5-a8fe-a0cf6c233712/kube-auth-proxy/0.log" Jul 23 12:12:36.112802 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:36.112699 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xzrb4_4ecc94e6-3546-4d27-85dc-973a6742c957/payload-pre-processing/0.log" Jul 23 12:12:36.135067 ip-10-0-128-30 kubenswrapper[2564]: I0723 12:12:36.135037 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-b8z4d_70b34262-a94d-432d-a805-8178c5fa049e/payload-processing/0.log"