Jul 15 14:33:49.473441 ip-10-0-141-12 systemd[1]: Starting Kubernetes Kubelet... Jul 15 14:33:49.879013 ip-10-0-141-12 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:49.879013 ip-10-0-141-12 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 14:33:49.879013 ip-10-0-141-12 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:49.879013 ip-10-0-141-12 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 14:33:49.879013 ip-10-0-141-12 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:49.881661 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.881561 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888640 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888660 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888667 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888671 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888675 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.888665 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888679 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888683 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888687 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888691 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888694 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888698 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888701 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888707 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888712 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888717 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888721 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888724 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888728 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888732 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888736 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888747 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888751 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888755 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888759 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.889010 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888763 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888767 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888771 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888775 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888779 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888783 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888787 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888791 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888795 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888798 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888802 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888807 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888810 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888814 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888818 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888823 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888827 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888831 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888835 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888841 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.889819 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888846 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888851 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888856 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888861 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888865 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888872 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888877 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888884 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888889 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888893 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888899 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888904 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888909 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888913 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888917 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888921 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888925 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888930 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888934 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888938 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.890693 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888942 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888947 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888951 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888955 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888959 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888963 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888967 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888973 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888978 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888982 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888986 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888990 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888994 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.888998 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889002 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889007 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889011 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889015 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889019 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889024 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.891600 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889028 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889032 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889036 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889041 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889044 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889049 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889069 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889074 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889078 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889083 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889087 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889100 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889104 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889108 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.889114 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890384 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890395 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890400 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890405 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.892443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890411 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890417 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890422 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890426 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890430 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890435 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890440 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890444 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890448 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890453 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890457 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890462 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890465 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890470 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890474 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890478 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890482 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890486 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890490 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890494 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.893315 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890498 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890502 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890506 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890511 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890516 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890521 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890525 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890529 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890532 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890536 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890540 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890544 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890549 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890553 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890558 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890562 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890566 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890570 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890574 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890578 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.893971 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890582 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890586 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890592 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890596 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890600 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890604 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890608 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890613 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890618 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890622 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890626 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890630 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890635 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890639 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890643 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890647 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890651 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890655 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890659 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890664 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.894495 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890668 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890673 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890679 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890685 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890689 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890697 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890702 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890707 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890711 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890715 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890719 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890724 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890728 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890732 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890739 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890746 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890750 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890754 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890759 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.895049 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890762 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890767 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890772 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890775 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890782 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890786 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890791 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890794 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890798 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890802 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890806 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890811 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890815 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890819 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890824 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.890828 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891482 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891491 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891495 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.895721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891501 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891505 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891511 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891515 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891518 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891522 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891526 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891530 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891533 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891542 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891546 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891553 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891558 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891562 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891567 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891572 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891577 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891581 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891585 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891590 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.896205 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891594 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891597 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891601 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891605 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891609 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891613 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891617 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891621 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891626 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891630 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891634 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891638 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891643 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891647 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891652 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891656 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891662 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891668 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891673 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.896661 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891678 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891683 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891687 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891693 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891699 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891704 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891711 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891716 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891720 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891725 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891729 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891734 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891738 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891742 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891746 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891750 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891754 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891758 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891762 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.897253 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891766 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891770 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891775 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891780 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891784 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891788 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891794 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891798 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891802 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891807 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891810 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891815 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891819 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891823 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891827 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891831 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891835 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891841 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891845 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891853 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.897723 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891857 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891861 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891866 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891870 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891874 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891878 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891882 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891886 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891890 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891894 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891898 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891901 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891905 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891910 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891914 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891918 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891922 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.891926 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892039 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892075 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892085 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 14:33:49.898217 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892093 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892099 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892105 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892111 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892117 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892123 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892127 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892133 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892139 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892145 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892153 2567 flags.go:64] FLAG: --cgroup-root="" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892158 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892162 2567 flags.go:64] FLAG: --client-ca-file="" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892167 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892171 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892179 2567 flags.go:64] FLAG: --cluster-domain="" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892184 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892189 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892194 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892199 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892205 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892210 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892216 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 14:33:49.898715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892221 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892226 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892230 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892235 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892242 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892246 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892257 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892262 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892267 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892271 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892276 2567 flags.go:64] FLAG: --enable-server="true" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892281 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892288 2567 flags.go:64] FLAG: --event-burst="100" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892292 2567 flags.go:64] FLAG: --event-qps="50" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892297 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892302 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892307 2567 flags.go:64] FLAG: --eviction-hard="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892313 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892318 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892324 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892330 2567 flags.go:64] FLAG: --eviction-soft="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892335 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892340 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892345 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892349 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 14:33:49.899271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892354 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892358 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892363 2567 flags.go:64] FLAG: --feature-gates="" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892368 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892373 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892378 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892384 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892389 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892393 2567 flags.go:64] FLAG: --help="false" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892398 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-141-12.ec2.internal" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892403 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892409 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892414 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892420 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892425 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892430 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892434 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892439 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892443 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892448 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892452 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892457 2567 flags.go:64] FLAG: --kube-reserved="" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892462 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892466 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 14:33:49.899904 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892471 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892476 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892483 2567 flags.go:64] FLAG: --lock-file="" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892488 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892493 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892498 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892507 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892511 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892516 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892521 2567 flags.go:64] FLAG: --logging-format="text" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892526 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892531 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892536 2567 flags.go:64] FLAG: --manifest-url="" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892540 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892547 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892552 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892559 2567 flags.go:64] FLAG: --max-pods="110" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892564 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892568 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892575 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892580 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892584 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892589 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892594 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892605 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 14:33:49.900555 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892610 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892614 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892619 2567 flags.go:64] FLAG: --pod-cidr="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892623 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892632 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892638 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892644 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892649 2567 flags.go:64] FLAG: --port="10250" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892654 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892662 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b10eb0687d281097" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892667 2567 flags.go:64] FLAG: --qos-reserved="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892673 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892678 2567 flags.go:64] FLAG: --register-node="true" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892683 2567 flags.go:64] FLAG: --register-with-taints="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892689 2567 flags.go:64] FLAG: --registry-burst="10" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892693 2567 flags.go:64] FLAG: --registry-qps="5" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892698 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892702 2567 flags.go:64] FLAG: --reserved-memory="" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892709 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892713 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892718 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892723 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892727 2567 flags.go:64] FLAG: --runonce="false" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892732 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892737 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 14:33:49.901278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892742 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892746 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892752 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892758 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892763 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892769 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892774 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892778 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892782 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892787 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892792 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892796 2567 flags.go:64] FLAG: --system-cgroups="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892800 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892809 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892814 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892818 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892830 2567 flags.go:64] FLAG: --tls-min-version="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892835 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892840 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892845 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892850 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892855 2567 flags.go:64] FLAG: --v="2" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892862 2567 flags.go:64] FLAG: --version="false" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892868 2567 flags.go:64] FLAG: --vmodule="" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892874 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 14:33:49.902243 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.892879 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893079 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893087 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893101 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893105 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893110 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893115 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893119 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893123 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893147 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893151 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893156 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893160 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893164 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893168 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893172 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893177 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893181 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893185 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893189 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.903302 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893193 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893197 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893202 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893208 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893212 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893218 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893222 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893227 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893231 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893235 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893239 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893243 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893248 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893252 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893256 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893261 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893265 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893269 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893273 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893277 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.904149 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893281 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893289 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893293 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893297 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893302 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893306 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893310 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893314 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893318 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893322 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893326 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893330 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893333 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893337 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893342 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893348 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893352 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893357 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893361 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.904838 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893368 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893375 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893380 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893384 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893388 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893393 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893399 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893405 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893410 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893415 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893419 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893424 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893431 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893437 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893443 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893448 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893452 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893456 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893460 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.905380 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893464 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893468 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893472 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893476 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893480 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893485 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893489 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893493 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893497 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893504 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893509 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893513 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893517 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893521 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893526 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893530 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893535 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893539 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893543 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893547 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.906176 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893551 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.893555 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.894546 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.902960 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.902983 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.903011 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.903026 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903131 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903141 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903146 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903152 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903156 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903162 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903166 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903171 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.906974 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903176 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903180 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903184 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903188 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903192 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903197 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903201 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903206 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903210 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903215 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903219 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903223 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903227 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903232 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903236 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903240 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903245 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903249 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903253 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903258 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.907385 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903262 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903265 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903270 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903274 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903278 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903283 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903287 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903291 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903297 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903303 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903308 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903314 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903320 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903326 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903330 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903335 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903340 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903344 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903349 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.907995 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903353 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903358 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903364 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903369 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903373 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903377 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903382 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903386 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903391 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903395 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903400 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903404 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903408 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903412 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903416 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903420 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903424 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903428 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903432 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903436 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.908467 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903440 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903444 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903448 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903453 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903457 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903461 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903465 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903469 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903473 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903477 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903481 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903485 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903490 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903494 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903498 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903503 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903508 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903512 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903516 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.908950 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903520 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903524 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903528 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903532 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903536 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903540 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903544 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903548 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903552 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903556 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903560 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903565 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903569 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.903577 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903787 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:49.909515 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903797 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903802 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903806 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903811 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903818 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903824 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903829 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903834 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903838 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903842 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903846 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903851 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903856 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903860 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903864 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903868 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903872 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903877 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903881 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903885 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:49.909877 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903889 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903893 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903897 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903901 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903905 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903910 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903916 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903922 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903926 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903931 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903935 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903939 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903943 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903948 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903952 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903957 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903961 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903965 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903968 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:49.910447 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903972 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903976 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903980 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903984 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903989 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903994 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.903998 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904002 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904006 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904010 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904015 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904019 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904023 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904027 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904032 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904036 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904040 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904046 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904051 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904074 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:49.910924 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904079 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904083 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904090 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904095 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904099 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904104 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904109 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904113 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904117 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904121 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904125 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904129 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904133 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904137 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904142 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904146 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904150 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904154 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904159 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904163 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:49.911471 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904167 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904170 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904174 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904178 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904183 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904187 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904191 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904195 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904199 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904206 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904210 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904214 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904218 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904222 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904228 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904232 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904237 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904241 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:49.912073 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:49.904245 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.904253 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.904469 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.907754 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.909300 2567 server.go:1019] "Starting client certificate rotation" Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.909406 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 14:33:49.912494 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.909448 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 14:33:49.912881 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.912863 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 14:33:49.914110 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.914087 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 14:33:49.923088 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.923069 2567 log.go:25] "Validated CRI v1 runtime API" Jul 15 14:33:49.925156 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.925119 2567 log.go:25] "Validated CRI v1 image API" Jul 15 14:33:49.926756 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.926740 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 14:33:49.929392 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.929371 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a8b5003a-9c8a-456b-ad33-dc63ac6bd402:/dev/nvme0n1p4 fe4370d5-18ed-4c46-9f18-03042d4d44b0:/dev/nvme0n1p3] Jul 15 14:33:49.929460 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.929392 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 14:33:49.934449 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.934320 2567 manager.go:216] Machine: {Timestamp:2026-07-15 14:33:49.932949296 +0000 UTC m=+0.342568566 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200688 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b4b71dd17ef0a99740ff17ee74c2c SystemUUID:ec2b4b71-dd17-ef0a-9974-0ff17ee74c2c BootID:26bdfde4-f2d3-4b90-968b-072233fe696a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:86:05:75:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:86:05:75:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:4f:0d:9a:a8:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 14:33:49.934449 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.934445 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 14:33:49.934556 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.934534 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 14:33:49.937964 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.937930 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 14:33:49.938118 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.937967 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-12.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 14:33:49.939410 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.939399 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 14:33:49.939442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.939413 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 14:33:49.939442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.939421 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 14:33:49.940110 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.940099 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 14:33:49.941467 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.941446 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 14:33:49.945087 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945049 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 15 14:33:49.945286 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945273 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 14:33:49.945328 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945311 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 15 14:33:49.945328 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945322 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 14:33:49.945393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945338 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 14:33:49.945393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945347 2567 kubelet.go:398] "Adding apiserver pod source" Jul 15 14:33:49.945393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945357 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 14:33:49.945728 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945709 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 14:33:49.945728 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.945727 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 14:33:49.947119 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.947101 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 14:33:49.948587 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.948573 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 14:33:49.950487 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950474 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 14:33:49.950525 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950503 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 14:33:49.950666 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950657 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 14:33:49.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950672 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 14:33:49.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950679 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 14:33:49.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950684 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 14:33:49.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950693 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 14:33:49.950792 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.950698 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 14:33:49.951296 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951287 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 14:33:49.951296 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951296 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 14:33:49.951352 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951304 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 14:33:49.951352 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951312 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 14:33:49.951352 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951324 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 14:33:49.951352 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951332 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 14:33:49.951456 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951355 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 14:33:49.951456 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.951360 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 14:33:49.955192 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.955078 2567 server.go:1268] "Started kubelet" Jul 15 14:33:49.955192 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.955164 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 14:33:49.955355 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.955179 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 14:33:49.955355 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.955315 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 14:33:49.955736 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.955719 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-12.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 14:33:49.955824 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.955741 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-12.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 14:33:49.955952 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.955924 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 14:33:49.956170 ip-10-0-141-12 systemd[1]: Started Kubernetes Kubelet. Jul 15 14:33:49.956890 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.956875 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 14:33:49.957745 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.957726 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 15 14:33:49.960329 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.960306 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 14:33:49.962757 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.962733 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 14:33:49.962853 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.962816 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 14:33:49.962907 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.962854 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:49.963458 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963441 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 14:33:49.963534 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963443 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 14:33:49.963534 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963480 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 14:33:49.963670 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963581 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 15 14:33:49.963670 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963588 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 15 14:33:49.963753 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963687 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 14:33:49.963753 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963700 2567 factory.go:55] Registering systemd factory Jul 15 14:33:49.963753 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963709 2567 factory.go:223] Registration of the systemd container factory successfully Jul 15 14:33:49.963992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963964 2567 factory.go:153] Registering CRI-O factory Jul 15 14:33:49.963992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.963980 2567 factory.go:223] Registration of the crio container factory successfully Jul 15 14:33:49.964167 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.964001 2567 factory.go:103] Registering Raw factory Jul 15 14:33:49.964167 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.964014 2567 manager.go:1196] Started watching for new ooms in manager Jul 15 14:33:49.964696 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.964683 2567 manager.go:349] Starting recovery of all containers Jul 15 14:33:49.964900 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.963888 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-12.ec2.internal.18c27d2be9ff45e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-12.ec2.internal,UID:ip-10-0-141-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-12.ec2.internal,},FirstTimestamp:2026-07-15 14:33:49.955024357 +0000 UTC m=+0.364643684,LastTimestamp:2026-07-15 14:33:49.955024357 +0000 UTC m=+0.364643684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-12.ec2.internal,}" Jul 15 14:33:49.970788 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.970741 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-12.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 14:33:49.970938 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.970915 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 14:33:49.975677 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.975553 2567 manager.go:354] Recovery completed Jul 15 14:33:49.980094 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.980079 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.983268 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983250 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hnqk" Jul 15 14:33:49.983347 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983334 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.983381 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983358 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.983381 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983373 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.983933 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983916 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 14:33:49.983933 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983931 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 14:33:49.984026 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983948 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 15 14:33:49.984026 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.983917 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.984746 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.984730 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.984790 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.984759 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.984790 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.984768 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.985250 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.985239 2567 policy_none.go:49] "None policy: Start" Jul 15 14:33:49.985283 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.985255 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 14:33:49.985283 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.985266 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 14:33:49.986386 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.986377 2567 policy_none.go:47] "Start" Jul 15 14:33:49.989719 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:49.989646 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-12.ec2.internal.18c27d2bebaf6367 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-12.ec2.internal,UID:ip-10-0-141-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-12.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-12.ec2.internal,},FirstTimestamp:2026-07-15 14:33:49.983343463 +0000 UTC m=+0.392962732,LastTimestamp:2026-07-15 14:33:49.983343463 +0000 UTC m=+0.392962732,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-12.ec2.internal,}" Jul 15 14:33:49.997309 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:49.997282 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hnqk" Jul 15 14:33:50.024657 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.024642 2567 manager.go:337] "Starting Device Plugin manager" Jul 15 14:33:50.024751 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.024678 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 14:33:50.024751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.024689 2567 server.go:87] "Starting device plugin registration server" Jul 15 14:33:50.024990 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.024977 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 14:33:50.025077 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.024993 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 14:33:50.025191 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.025168 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 14:33:50.025277 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.025260 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 14:33:50.025277 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.025272 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 14:33:50.026455 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.026432 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 14:33:50.026522 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.026482 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.126131 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.126104 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:50.127256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.127239 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:50.127325 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.127271 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:50.127325 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.127287 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:50.127325 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.127315 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.140454 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.140398 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.140454 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.140422 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-12.ec2.internal\": node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.156743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.156714 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 14:33:50.157984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.157968 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 14:33:50.158050 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.158005 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 14:33:50.158050 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.158026 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 14:33:50.158249 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.158075 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 14:33:50.158544 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.158525 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.164304 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.164288 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:50.258190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.258150 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal"] Jul 15 14:33:50.258367 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.258238 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:50.258646 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.258625 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.259222 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.259204 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:50.259328 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.259235 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:50.259328 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.259250 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:50.260745 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.260727 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:50.260849 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.260823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.260849 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.260849 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:50.261524 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261510 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:50.261618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261540 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:50.261618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261554 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:50.261618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261513 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:50.261763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261614 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:50.261763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.261654 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:50.262717 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.262703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.262800 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.262734 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:50.263394 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.263380 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:50.263466 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.263402 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:50.263466 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.263412 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:50.264764 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.264749 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.264827 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.264771 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.264827 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.264791 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17256dde85068a9f5597774cee525f87-config\") pod \"kube-apiserver-proxy-ip-10-0-141-12.ec2.internal\" (UID: \"17256dde85068a9f5597774cee525f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.291874 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.291850 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-12.ec2.internal\" not found" node="ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.296655 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.296639 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-12.ec2.internal\" not found" node="ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.359184 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.359155 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.365498 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.365549 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.365549 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365521 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17256dde85068a9f5597774cee525f87-config\") pod \"kube-apiserver-proxy-ip-10-0-141-12.ec2.internal\" (UID: \"17256dde85068a9f5597774cee525f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.365610 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365547 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17256dde85068a9f5597774cee525f87-config\") pod \"kube-apiserver-proxy-ip-10-0-141-12.ec2.internal\" (UID: \"17256dde85068a9f5597774cee525f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.365610 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.365610 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.365580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48ac1fb108d098105620145183d2aa95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal\" (UID: \"48ac1fb108d098105620145183d2aa95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.460201 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.460133 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.560688 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.560656 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.594169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.594137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.599106 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.599084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:50.661042 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.661001 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.761637 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.761562 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.847820 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.847791 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:50.862473 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.862440 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:50.908865 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.908833 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 14:33:50.909513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.909024 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 14:33:50.909513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:50.909025 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 14:33:50.962989 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:50.962951 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:51.000552 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.000498 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 14:28:50 +0000 UTC" deadline="2028-01-23 01:58:07.832775988 +0000 UTC" Jul 15 14:33:51.000552 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.000544 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13355h24m16.832235443s" Jul 15 14:33:51.063863 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:51.063838 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-12.ec2.internal\" not found" Jul 15 14:33:51.123510 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.123485 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:51.150123 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:51.150096 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17256dde85068a9f5597774cee525f87.slice/crio-6495d85a9b8de1e37fe58ed6dbe538f2e9c0e9c7d0b2dfc23e90730a23b36db7 WatchSource:0}: Error finding container 6495d85a9b8de1e37fe58ed6dbe538f2e9c0e9c7d0b2dfc23e90730a23b36db7: Status 404 returned error can't find the container with id 6495d85a9b8de1e37fe58ed6dbe538f2e9c0e9c7d0b2dfc23e90730a23b36db7 Jul 15 14:33:51.155605 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.155581 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:33:51.161829 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.161789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" event={"ID":"17256dde85068a9f5597774cee525f87","Type":"ContainerStarted","Data":"6495d85a9b8de1e37fe58ed6dbe538f2e9c0e9c7d0b2dfc23e90730a23b36db7"} Jul 15 14:33:51.163925 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.163911 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" Jul 15 14:33:51.180241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.180205 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 14:33:51.182202 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.182185 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" Jul 15 14:33:51.193754 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.193734 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 14:33:51.245657 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.244980 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:51.947148 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.947046 2567 apiserver.go:52] "Watching apiserver" Jul 15 14:33:51.952712 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.952686 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 14:33:51.953144 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.953116 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-9crlm","openshift-dns/node-resolver-77j4x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal","openshift-multus/multus-69q2k","openshift-multus/multus-additional-cni-plugins-t9474","openshift-network-operator/iptables-alerter-w5f8v","kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb","openshift-image-registry/node-ca-nzwnl","openshift-multus/network-metrics-daemon-9z5t2","openshift-network-diagnostics/network-check-target-rwcbt","openshift-ovn-kubernetes/ovnkube-node-v25jf","kube-system/global-pull-secret-syncer-864x5","kube-system/konnectivity-agent-p67nb"] Jul 15 14:33:51.954970 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.954927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:51.955097 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:51.955025 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:33:51.956558 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.956524 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:51.956670 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:51.956603 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:33:51.957827 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.957806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:51.959097 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.959076 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.960488 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.960467 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.960795 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.960773 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 14:33:51.961071 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.961035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 14:33:51.961789 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.961771 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:51.962292 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.962272 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 14:33:51.963132 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.963113 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 14:33:51.963248 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.963214 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:51.964805 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.964782 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 14:33:51.964933 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.964813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.964992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.964947 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thpdx\"" Jul 15 14:33:51.966741 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.966418 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:51.966741 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.966594 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mwb7r\"" Jul 15 14:33:51.967179 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.967156 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.967899 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.967878 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:51.969258 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.969238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:51.969376 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:51.969353 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:33:51.971008 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.970986 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.971118 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.971036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:51.974558 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974530 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d13942e3-ce89-4455-9a17-7681469756ce-serviceca\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:51.974652 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974567 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsmj\" (UniqueName: \"kubernetes.io/projected/a736a3ad-c62f-45ec-974e-2902d5c4ee62-kube-api-access-rtsmj\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:51.974652 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-kubelet-config\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:51.974652 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-cnibin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-cni-binary-copy\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-kubelet\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974698 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-multus-certs\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.974798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2s7\" (UniqueName: \"kubernetes.io/projected/88524ec0-8260-48f6-adbe-a9fa04257b25-kube-api-access-9d2s7\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/792ca10f-5da9-4995-a253-91b8609a1585-iptables-alerter-script\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-socket-dir-parent\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-conf-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8k2w\" (UniqueName: \"kubernetes.io/projected/792ca10f-5da9-4995-a253-91b8609a1585-kube-api-access-g8k2w\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-device-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.974992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-konnectivity-ca\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:51.975047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.975451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975014 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.975451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975108 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dtmp6\"" Jul 15 14:33:51.975451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975209 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 14:33:51.976064 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.975938 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 14:33:51.976064 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.976059 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c2x9z\"" Jul 15 14:33:51.976225 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.976062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.976627 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.976377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh4m\" (UniqueName: \"kubernetes.io/projected/62e20b9a-ec6b-4119-8892-4666e75413fb-kube-api-access-lrh4m\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.976723 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.976640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-system-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-netns\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977468 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-bin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-daemon-config\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-agent-certs\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977567 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-os-release\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/792ca10f-5da9-4995-a253-91b8609a1585-host-slash\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977655 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4v7v\" (UniqueName: \"kubernetes.io/projected/d13942e3-ce89-4455-9a17-7681469756ce-kube-api-access-r4v7v\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdklt\" (UniqueName: \"kubernetes.io/projected/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kube-api-access-wdklt\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-dbus\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-multus\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-cnibin\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.980905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-sys-fs\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.977962 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-os-release\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978305 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-hostroot\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d13942e3-ce89-4455-9a17-7681469756ce-host\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978384 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-socket-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-registration-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978616 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-k8s-cni-cncf-io\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-etc-kubernetes\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978832 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.978951 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8c9t\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979064 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979329 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979511 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979556 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979717 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k6tnr\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979767 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979977 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980070 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980150 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980322 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980385 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.981715 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980443 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.982766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.979517 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-km7qp\"" Jul 15 14:33:51.982766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980541 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 14:33:51.982766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980648 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wsxn\"" Jul 15 14:33:51.982766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.980742 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xqhdf\"" Jul 15 14:33:51.982766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.982041 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 14:33:51.993232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:51.993209 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 14:33:52.005511 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.005488 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5mwm" Jul 15 14:33:52.026161 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.026139 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5mwm" Jul 15 14:33:52.064963 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.064932 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 14:33:52.080075 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-env-overrides\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.080075 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3f1225f-4a90-4e01-847d-2fe17310bada-ovn-node-metrics-cert\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-script-lib\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-tmp\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-var-lib-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-netns\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.080334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-bin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-daemon-config\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/792ca10f-5da9-4995-a253-91b8609a1585-host-slash\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-var-lib-kubelet\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-bin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-netns\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080510 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/792ca10f-5da9-4995-a253-91b8609a1585-host-slash\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-log-socket\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.080623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4v7v\" (UniqueName: \"kubernetes.io/projected/d13942e3-ce89-4455-9a17-7681469756ce-kube-api-access-r4v7v\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.080672 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-dbus\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.080767 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.580730899 +0000 UTC m=+2.990350179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-cnibin\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-kubelet\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-netns\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-dbus\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-etc-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080907 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-cnibin\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080908 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-config\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-sys-fs\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.080988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-hostroot\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-conf\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-bin\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716f201f-0bc0-4fb7-92a4-124c180171bd-hosts-file\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.081220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081090 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysconfig\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-hostroot\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081107 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-sys-fs\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-slash\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081197 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsmj\" (UniqueName: \"kubernetes.io/projected/a736a3ad-c62f-45ec-974e-2902d5c4ee62-kube-api-access-rtsmj\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-multus-certs\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-daemon-config\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2s7\" (UniqueName: \"kubernetes.io/projected/88524ec0-8260-48f6-adbe-a9fa04257b25-kube-api-access-9d2s7\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-sys\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrbm\" (UniqueName: \"kubernetes.io/projected/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-kube-api-access-nqrbm\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-modprobe-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-kubernetes\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081410 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-tuned\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081434 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-multus-certs\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.081985 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9v82\" (UniqueName: \"kubernetes.io/projected/716f201f-0bc0-4fb7-92a4-124c180171bd-kube-api-access-f9v82\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-device-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081503 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-konnectivity-ca\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-device-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh4m\" (UniqueName: \"kubernetes.io/projected/62e20b9a-ec6b-4119-8892-4666e75413fb-kube-api-access-lrh4m\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.081636 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-system-cni-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-run\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.081708 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.581691733 +0000 UTC m=+2.991310995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-system-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-agent-certs\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-os-release\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-systemd\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081852 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-system-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.082796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-lib-modules\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-ovn\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081917 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081943 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdklt\" (UniqueName: \"kubernetes.io/projected/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kube-api-access-wdklt\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-multus\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-netd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f201f-0bc0-4fb7-92a4-124c180171bd-tmp-dir\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-os-release\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-host\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082187 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-konnectivity-ca\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d13942e3-ce89-4455-9a17-7681469756ce-host\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082205 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-socket-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.083589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-registration-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-os-release\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082260 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-cni-multus\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d13942e3-ce89-4455-9a17-7681469756ce-host\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.081918 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-os-release\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-k8s-cni-cncf-io\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082382 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-registration-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-cni-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-etc-kubernetes\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-etc-kubernetes\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-run-k8s-cni-cncf-io\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-socket-dir\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-node-log\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d13942e3-ce89-4455-9a17-7681469756ce-serviceca\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-kubelet-config\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-cnibin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-cni-binary-copy\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.084372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-kubelet\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-cnibin\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62e20b9a-ec6b-4119-8892-4666e75413fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082665 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/792ca10f-5da9-4995-a253-91b8609a1585-iptables-alerter-script\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw6w\" (UniqueName: \"kubernetes.io/projected/f3f1225f-4a90-4e01-847d-2fe17310bada-kube-api-access-zfw6w\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-socket-dir-parent\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-conf-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8k2w\" (UniqueName: \"kubernetes.io/projected/792ca10f-5da9-4995-a253-91b8609a1585-kube-api-access-g8k2w\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-systemd-units\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-systemd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.082953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-conf-dir\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-multus-socket-dir-parent\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d13942e3-ce89-4455-9a17-7681469756ce-serviceca\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62e20b9a-ec6b-4119-8892-4666e75413fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.085122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a835c6a-ccd5-40db-b075-a1437d022fc1-kubelet-config\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.085922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88524ec0-8260-48f6-adbe-a9fa04257b25-cni-binary-copy\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.083191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88524ec0-8260-48f6-adbe-a9fa04257b25-host-var-lib-kubelet\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.085922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.084137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/792ca10f-5da9-4995-a253-91b8609a1585-iptables-alerter-script\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.085922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.084477 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:52.086478 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.086455 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae03a625-f5e7-4b8f-9b99-4c704f4c9366-agent-certs\") pod \"konnectivity-agent-p67nb\" (UID: \"ae03a625-f5e7-4b8f-9b99-4c704f4c9366\") " pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:52.097838 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.097770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4v7v\" (UniqueName: \"kubernetes.io/projected/d13942e3-ce89-4455-9a17-7681469756ce-kube-api-access-r4v7v\") pod \"node-ca-nzwnl\" (UID: \"d13942e3-ce89-4455-9a17-7681469756ce\") " pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.099881 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.099852 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2s7\" (UniqueName: \"kubernetes.io/projected/88524ec0-8260-48f6-adbe-a9fa04257b25-kube-api-access-9d2s7\") pod \"multus-69q2k\" (UID: \"88524ec0-8260-48f6-adbe-a9fa04257b25\") " pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.100023 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.099989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8k2w\" (UniqueName: \"kubernetes.io/projected/792ca10f-5da9-4995-a253-91b8609a1585-kube-api-access-g8k2w\") pod \"iptables-alerter-w5f8v\" (UID: \"792ca10f-5da9-4995-a253-91b8609a1585\") " pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.100719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.100457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh4m\" (UniqueName: \"kubernetes.io/projected/62e20b9a-ec6b-4119-8892-4666e75413fb-kube-api-access-lrh4m\") pod \"multus-additional-cni-plugins-t9474\" (UID: \"62e20b9a-ec6b-4119-8892-4666e75413fb\") " pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.100917 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.100898 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsmj\" (UniqueName: \"kubernetes.io/projected/a736a3ad-c62f-45ec-974e-2902d5c4ee62-kube-api-access-rtsmj\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:52.101426 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.101408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdklt\" (UniqueName: \"kubernetes.io/projected/8c5e33d9-83cd-41df-80a4-f606b2e1e8fd-kube-api-access-wdklt\") pod \"aws-ebs-csi-driver-node-phxmb\" (UID: \"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.166914 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.166867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" event={"ID":"48ac1fb108d098105620145183d2aa95","Type":"ContainerStarted","Data":"e40243ee6f87ebad87a825e6b349be517a8277e5bf8b7509fbe93195739c2db0"} Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-kubernetes\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-tuned\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9v82\" (UniqueName: \"kubernetes.io/projected/716f201f-0bc0-4fb7-92a4-124c180171bd-kube-api-access-f9v82\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-kubernetes\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-run\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-systemd\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-lib-modules\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-ovn\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-netd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f201f-0bc0-4fb7-92a4-124c180171bd-tmp-dir\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-host\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-node-log\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw6w\" (UniqueName: \"kubernetes.io/projected/f3f1225f-4a90-4e01-847d-2fe17310bada-kube-api-access-zfw6w\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184565 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-systemd-units\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184588 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-systemd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184610 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-env-overrides\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.185169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3f1225f-4a90-4e01-847d-2fe17310bada-ovn-node-metrics-cert\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-script-lib\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-tmp\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-var-lib-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-var-lib-kubelet\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184761 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-node-log\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-log-socket\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-kubelet\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-run\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.184894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-systemd\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-lib-modules\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-ovn\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-netd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f201f-0bc0-4fb7-92a4-124c180171bd-tmp-dir\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-host\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-script-lib\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185614 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-kubelet\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.186236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.185667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-env-overrides\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-systemd-units\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186337 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-systemd\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-netns\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-etc-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-config\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-conf\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-bin\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716f201f-0bc0-4fb7-92a4-124c180171bd-hosts-file\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysconfig\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-slash\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186636 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-var-lib-kubelet\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186644 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-log-socket\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.187171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-run-netns\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-sys\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-etc-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-run-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186916 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysconfig\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187107 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-sysctl-conf\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-cni-bin\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-tmp\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187252 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716f201f-0bc0-4fb7-92a4-124c180171bd-hosts-file\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187269 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-host-slash\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.186583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3f1225f-4a90-4e01-847d-2fe17310bada-var-lib-openvswitch\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-tuned\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrbm\" (UniqueName: \"kubernetes.io/projected/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-kube-api-access-nqrbm\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-modprobe-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-sys\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-etc-modprobe-d\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.188013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.187426 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3f1225f-4a90-4e01-847d-2fe17310bada-ovnkube-config\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.189701 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.189681 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3f1225f-4a90-4e01-847d-2fe17310bada-ovn-node-metrics-cert\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.194044 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.193694 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:52.194044 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.193724 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:52.194044 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.193740 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:52.194044 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.193818 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.693798384 +0000 UTC m=+3.103417647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:52.194397 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.194300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9v82\" (UniqueName: \"kubernetes.io/projected/716f201f-0bc0-4fb7-92a4-124c180171bd-kube-api-access-f9v82\") pod \"node-resolver-77j4x\" (UID: \"716f201f-0bc0-4fb7-92a4-124c180171bd\") " pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.196027 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.195982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw6w\" (UniqueName: \"kubernetes.io/projected/f3f1225f-4a90-4e01-847d-2fe17310bada-kube-api-access-zfw6w\") pod \"ovnkube-node-v25jf\" (UID: \"f3f1225f-4a90-4e01-847d-2fe17310bada\") " pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.200967 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.200889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrbm\" (UniqueName: \"kubernetes.io/projected/53bc20ce-8a10-415f-80e4-f4a39f34dc2b-kube-api-access-nqrbm\") pod \"tuned-9crlm\" (UID: \"53bc20ce-8a10-415f-80e4-f4a39f34dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.269909 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.269872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:33:52.278028 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.278002 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69q2k" Jul 15 14:33:52.289177 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.289148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9474" Jul 15 14:33:52.296300 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.296273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5f8v" Jul 15 14:33:52.303465 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.303432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nzwnl" Jul 15 14:33:52.311816 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.311778 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" Jul 15 14:33:52.320001 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.319970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77j4x" Jul 15 14:33:52.329187 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.329158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9crlm" Jul 15 14:33:52.335360 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.335335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:33:52.589550 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.589461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:52.589689 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.589549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:52.589689 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.589626 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.589689 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.589638 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.589820 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.589700 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:53.589684832 +0000 UTC m=+3.999304104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.589820 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.589716 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:53.589708896 +0000 UTC m=+3.999328154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.623821 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.623785 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:52.790981 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:52.790934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:52.791220 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.791168 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:52.791220 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.791222 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:52.791419 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.791236 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:52.791419 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:52.791303 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:53.791285728 +0000 UTC m=+4.200904987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:53.029371 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.029260 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 14:28:52 +0000 UTC" deadline="2028-03-24 09:21:30.68771456 +0000 UTC" Jul 15 14:33:53.029371 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.029309 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14826h47m37.658410859s" Jul 15 14:33:53.159305 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.159266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:53.159502 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.159266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:53.159502 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.159413 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:33:53.159659 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.159550 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:33:53.219266 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.219221 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13942e3_ce89_4455_9a17_7681469756ce.slice/crio-d33ff7d9b58bb971ec6b50413da1c4e543290241c1442cb6e71c3ded2370337f WatchSource:0}: Error finding container d33ff7d9b58bb971ec6b50413da1c4e543290241c1442cb6e71c3ded2370337f: Status 404 returned error can't find the container with id d33ff7d9b58bb971ec6b50413da1c4e543290241c1442cb6e71c3ded2370337f Jul 15 14:33:53.222513 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.222418 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bc20ce_8a10_415f_80e4_f4a39f34dc2b.slice/crio-74a1d7b8fcc86a4fb9132550330ae4119dd7ea65ebfdaaacb619cb03ae4a23da WatchSource:0}: Error finding container 74a1d7b8fcc86a4fb9132550330ae4119dd7ea65ebfdaaacb619cb03ae4a23da: Status 404 returned error can't find the container with id 74a1d7b8fcc86a4fb9132550330ae4119dd7ea65ebfdaaacb619cb03ae4a23da Jul 15 14:33:53.224989 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.224962 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5e33d9_83cd_41df_80a4_f606b2e1e8fd.slice/crio-76d20442e72ed07ef1497e49ea49c60d8d3f98f8c7a0251a193a71ec4e3730a0 WatchSource:0}: Error finding container 76d20442e72ed07ef1497e49ea49c60d8d3f98f8c7a0251a193a71ec4e3730a0: Status 404 returned error can't find the container with id 76d20442e72ed07ef1497e49ea49c60d8d3f98f8c7a0251a193a71ec4e3730a0 Jul 15 14:33:53.228762 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.228738 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae03a625_f5e7_4b8f_9b99_4c704f4c9366.slice/crio-9ffd1a0af14e5a4926cf188d8c306709fc6ec0e464a09d523499a1d0548bdb5e WatchSource:0}: Error finding container 9ffd1a0af14e5a4926cf188d8c306709fc6ec0e464a09d523499a1d0548bdb5e: Status 404 returned error can't find the container with id 9ffd1a0af14e5a4926cf188d8c306709fc6ec0e464a09d523499a1d0548bdb5e Jul 15 14:33:53.230411 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.230352 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88524ec0_8260_48f6_adbe_a9fa04257b25.slice/crio-0b9d0f06b3ff006825e6889e343c081a5d00332e7b88d2da723f6c6bec563768 WatchSource:0}: Error finding container 0b9d0f06b3ff006825e6889e343c081a5d00332e7b88d2da723f6c6bec563768: Status 404 returned error can't find the container with id 0b9d0f06b3ff006825e6889e343c081a5d00332e7b88d2da723f6c6bec563768 Jul 15 14:33:53.231443 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.231419 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f1225f_4a90_4e01_847d_2fe17310bada.slice/crio-b7fc40181320e9c5648474a1a1abca485e8d65d00c7d802da2e36cdd4cf334c3 WatchSource:0}: Error finding container b7fc40181320e9c5648474a1a1abca485e8d65d00c7d802da2e36cdd4cf334c3: Status 404 returned error can't find the container with id b7fc40181320e9c5648474a1a1abca485e8d65d00c7d802da2e36cdd4cf334c3 Jul 15 14:33:53.236866 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:33:53.236787 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792ca10f_5da9_4995_a253_91b8609a1585.slice/crio-afdd51ade6328b27d59cf1caf2e109a290bcc318792a5159ea7f9ea2a088186c WatchSource:0}: Error finding container afdd51ade6328b27d59cf1caf2e109a290bcc318792a5159ea7f9ea2a088186c: Status 404 returned error can't find the container with id afdd51ade6328b27d59cf1caf2e109a290bcc318792a5159ea7f9ea2a088186c Jul 15 14:33:53.597373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.597006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:53.597373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.597362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:53.597566 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.597212 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:53.597566 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.597451 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:55.59743356 +0000 UTC m=+6.007052831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:53.597566 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.597483 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:53.597566 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.597540 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:55.597525439 +0000 UTC m=+6.007144699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:53.799417 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:53.799358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:53.799598 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.799570 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:53.799676 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.799605 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:53.799676 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.799620 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:53.799782 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:53.799696 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:55.799675926 +0000 UTC m=+6.209295187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:54.029526 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.029451 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 14:28:52 +0000 UTC" deadline="2028-03-09 11:16:22.224212456 +0000 UTC" Jul 15 14:33:54.029526 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.029522 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14468h42m28.194695186s" Jul 15 14:33:54.161538 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.160921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:54.161538 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:54.161068 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:33:54.177896 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.177833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"b7fc40181320e9c5648474a1a1abca485e8d65d00c7d802da2e36cdd4cf334c3"} Jul 15 14:33:54.186726 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.186684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69q2k" event={"ID":"88524ec0-8260-48f6-adbe-a9fa04257b25","Type":"ContainerStarted","Data":"0b9d0f06b3ff006825e6889e343c081a5d00332e7b88d2da723f6c6bec563768"} Jul 15 14:33:54.196845 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.196799 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" event={"ID":"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd","Type":"ContainerStarted","Data":"76d20442e72ed07ef1497e49ea49c60d8d3f98f8c7a0251a193a71ec4e3730a0"} Jul 15 14:33:54.199064 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.199022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nzwnl" event={"ID":"d13942e3-ce89-4455-9a17-7681469756ce","Type":"ContainerStarted","Data":"d33ff7d9b58bb971ec6b50413da1c4e543290241c1442cb6e71c3ded2370337f"} Jul 15 14:33:54.205705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.205668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5f8v" event={"ID":"792ca10f-5da9-4995-a253-91b8609a1585","Type":"ContainerStarted","Data":"afdd51ade6328b27d59cf1caf2e109a290bcc318792a5159ea7f9ea2a088186c"} Jul 15 14:33:54.209177 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.208751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p67nb" event={"ID":"ae03a625-f5e7-4b8f-9b99-4c704f4c9366","Type":"ContainerStarted","Data":"9ffd1a0af14e5a4926cf188d8c306709fc6ec0e464a09d523499a1d0548bdb5e"} Jul 15 14:33:54.211188 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.210917 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9crlm" event={"ID":"53bc20ce-8a10-415f-80e4-f4a39f34dc2b","Type":"ContainerStarted","Data":"74a1d7b8fcc86a4fb9132550330ae4119dd7ea65ebfdaaacb619cb03ae4a23da"} Jul 15 14:33:54.216421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.215709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" event={"ID":"17256dde85068a9f5597774cee525f87","Type":"ContainerStarted","Data":"32e322967f54dd894379274a3017e659cf71208f9e10e259eb06f702c85706ed"} Jul 15 14:33:54.226098 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.225457 2567 generic.go:356] "Generic (PLEG): container finished" podID="48ac1fb108d098105620145183d2aa95" containerID="9d47a195ba89563ebf51aa338cf3a6875702b70bcbd843aa5ea889476c9fc58a" exitCode=0 Jul 15 14:33:54.226098 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.225601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" event={"ID":"48ac1fb108d098105620145183d2aa95","Type":"ContainerDied","Data":"9d47a195ba89563ebf51aa338cf3a6875702b70bcbd843aa5ea889476c9fc58a"} Jul 15 14:33:54.228327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.228296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerStarted","Data":"767c46b77b91465a2ef46de3353a2a40df1ca47a0a50b885acd2ef8dd5e32270"} Jul 15 14:33:54.232035 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.231885 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77j4x" event={"ID":"716f201f-0bc0-4fb7-92a4-124c180171bd","Type":"ContainerStarted","Data":"7f9fbb64248496978d436f46047fb041300fa23d4353af2ffbd59e07231771f3"} Jul 15 14:33:54.234182 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:54.234078 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-12.ec2.internal" podStartSLOduration=3.234059958 podStartE2EDuration="3.234059958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:33:54.233448865 +0000 UTC m=+4.643068147" watchObservedRunningTime="2026-07-15 14:33:54.234059958 +0000 UTC m=+4.643679239" Jul 15 14:33:55.158895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.158853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:55.159430 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.159003 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:33:55.160716 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.159595 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:55.160716 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.159714 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:33:55.255462 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.255379 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" event={"ID":"48ac1fb108d098105620145183d2aa95","Type":"ContainerStarted","Data":"4346690c9fbd4aba370607f51e84e6b0987ca0d2912dc5e31cd440f17f134162"} Jul 15 14:33:55.615250 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.615159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:55.615250 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.615226 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:55.615496 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.615360 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:55.615496 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.615427 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:59.615405544 +0000 UTC m=+10.025024805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:55.615838 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.615818 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:55.615917 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.615875 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:59.615858738 +0000 UTC m=+10.025478002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:55.817073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:55.816334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:55.817073 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.816568 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:55.817073 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.816589 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:55.817073 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.816606 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:55.817073 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:55.816668 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:59.816649223 +0000 UTC m=+10.226268483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:56.159334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:56.159303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:56.159830 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:56.159442 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:33:57.158492 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:57.158452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:57.158679 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:57.158604 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:33:57.159066 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:57.159016 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:57.159246 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:57.159160 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:33:58.160385 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:58.160273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:58.160868 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:58.160447 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:33:59.159337 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:59.159294 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:59.159538 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:59.159307 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:59.159538 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.159444 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:33:59.159665 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.159543 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:59.652251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:59.652321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.652486 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.652556 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:07.652534683 +0000 UTC m=+18.062153947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.652672 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:59.652951 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.652747 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:07.652727521 +0000 UTC m=+18.062346787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:59.854448 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:33:59.854405 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:33:59.854639 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.854606 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:59.854639 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.854636 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:59.854728 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.854650 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:59.854728 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:33:59.854711 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:07.854692127 +0000 UTC m=+18.264311387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:00.159435 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:00.159397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:00.159607 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:00.159482 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:01.158608 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:01.158569 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:01.159104 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:01.158569 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:01.159104 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:01.158712 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:01.159104 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:01.158842 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:02.158897 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:02.158859 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:02.159360 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:02.159008 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:03.159149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:03.159114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:03.159149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:03.159139 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:03.159662 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:03.159254 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:03.159662 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:03.159378 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:04.160803 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:04.160778 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:04.161181 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:04.160886 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:05.158402 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:05.158367 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:05.158402 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:05.158399 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:05.158628 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:05.158532 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:05.158687 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:05.158632 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:06.162463 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:06.162432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:06.162867 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:06.162565 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:07.158778 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:07.158733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:07.158978 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:07.158754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:07.158978 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.158877 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:07.158978 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.158930 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:07.715370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:07.715330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:07.715849 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:07.715393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:07.715849 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.715511 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:07.715849 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.715577 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:23.715558714 +0000 UTC m=+34.125177975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:07.716043 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.715987 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:07.716043 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.716038 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:23.71602233 +0000 UTC m=+34.125641591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:07.917652 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:07.917619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:07.917822 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.917800 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:34:07.917864 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.917830 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:34:07.917864 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.917843 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:07.917918 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:07.917894 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:23.917881555 +0000 UTC m=+34.327500811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:08.158442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:08.158359 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:08.158602 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:08.158507 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:09.158779 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:09.158739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:09.159251 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:09.158789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:09.159251 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:09.158881 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:09.159251 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:09.159018 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:10.159319 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:10.159265 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:10.159714 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:10.159390 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:11.158986 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:11.158949 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:11.159170 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:11.158949 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:11.159170 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:11.159112 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:11.159276 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:11.159172 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:12.160021 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.159769 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:12.160435 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:12.160155 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:12.297684 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.297622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerStarted","Data":"8a816ecd31bd83532dfc69c5a6be79219e670dfe6fe638d2bb5f2f6d9abbe2f4"} Jul 15 14:34:12.300262 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.300228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77j4x" event={"ID":"716f201f-0bc0-4fb7-92a4-124c180171bd","Type":"ContainerStarted","Data":"c77cf9fbf5b19f0052ad180329cfecabec0b7719b41aa4523d253824d9a794e5"} Jul 15 14:34:12.302112 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.302090 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"0afc1a9c4801649fe254055695e2d2ba981ac8706e2672706135c06077f8fbb0"} Jul 15 14:34:12.304222 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.304134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69q2k" event={"ID":"88524ec0-8260-48f6-adbe-a9fa04257b25","Type":"ContainerStarted","Data":"4bdfe90f201792bb937330a4d763bcb6858d79021ce3abd032bd9dec144d77e6"} Jul 15 14:34:12.306614 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.306583 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" event={"ID":"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd","Type":"ContainerStarted","Data":"f5bf33b38d9d660a289018db3b7a21dda9ecf25054d0172d0b7ce921e8e43009"} Jul 15 14:34:12.308358 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.308335 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nzwnl" event={"ID":"d13942e3-ce89-4455-9a17-7681469756ce","Type":"ContainerStarted","Data":"af392143e8c4da04f3e93a6c199c886d78b6166f6aa6986925be61ce56f25890"} Jul 15 14:34:12.311091 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.310646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p67nb" event={"ID":"ae03a625-f5e7-4b8f-9b99-4c704f4c9366","Type":"ContainerStarted","Data":"27be36174ee8c9bac978f36c4a5000af597f118c36b969ff657830a54d0c5d7b"} Jul 15 14:34:12.312573 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.312552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9crlm" event={"ID":"53bc20ce-8a10-415f-80e4-f4a39f34dc2b","Type":"ContainerStarted","Data":"148ddf1319c9d2838429372a8c1b1db99e8e9f6c9d3f6eecf850750992100e06"} Jul 15 14:34:12.326235 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.326181 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-12.ec2.internal" podStartSLOduration=21.326165429 podStartE2EDuration="21.326165429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:33:55.275954659 +0000 UTC m=+5.685573942" watchObservedRunningTime="2026-07-15 14:34:12.326165429 +0000 UTC m=+22.735784706" Jul 15 14:34:12.347747 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.347691 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9crlm" podStartSLOduration=3.470061905 podStartE2EDuration="22.347668372s" totalImagesPullingTime="18.877606467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.345624634 +0000 UTC m=+22.755243963" watchObservedRunningTime="2026-07-15 14:34:12.347668372 +0000 UTC m=+22.757287652" Jul 15 14:34:12.379142 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.379088 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p67nb" podStartSLOduration=7.786805251 podStartE2EDuration="22.379067746s" totalImagesPullingTime="14.592262495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.378949406 +0000 UTC m=+22.788568684" watchObservedRunningTime="2026-07-15 14:34:12.379067746 +0000 UTC m=+22.788687018" Jul 15 14:34:12.402279 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.402225 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-69q2k" podStartSLOduration=3.480011969 podStartE2EDuration="22.402210429s" totalImagesPullingTime="18.92219846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.402159749 +0000 UTC m=+22.811779027" watchObservedRunningTime="2026-07-15 14:34:12.402210429 +0000 UTC m=+22.811829706" Jul 15 14:34:12.421349 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.421285 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nzwnl" podStartSLOduration=3.5633056979999997 podStartE2EDuration="22.421261104s" totalImagesPullingTime="18.857955406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.420253133 +0000 UTC m=+22.829872416" watchObservedRunningTime="2026-07-15 14:34:12.421261104 +0000 UTC m=+22.830880408" Jul 15 14:34:12.438888 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:12.438848 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-77j4x" podStartSLOduration=3.59379149 podStartE2EDuration="22.438832126s" totalImagesPullingTime="18.845040636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.438221726 +0000 UTC m=+22.847841001" watchObservedRunningTime="2026-07-15 14:34:12.438832126 +0000 UTC m=+22.848451403" Jul 15 14:34:13.159012 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.158832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:13.159193 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.158852 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:13.159193 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:13.159107 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:13.159193 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:13.159177 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:13.316362 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.316277 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="8a816ecd31bd83532dfc69c5a6be79219e670dfe6fe638d2bb5f2f6d9abbe2f4" exitCode=0 Jul 15 14:34:13.317221 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.316358 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"8a816ecd31bd83532dfc69c5a6be79219e670dfe6fe638d2bb5f2f6d9abbe2f4"} Jul 15 14:34:13.319178 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319143 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:34:13.319544 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319525 2567 generic.go:356] "Generic (PLEG): container finished" podID="f3f1225f-4a90-4e01-847d-2fe17310bada" containerID="17e106c1c1a63749db7ce8db347d7584d3069c9dcc21fa9a84aa6c8d803817d4" exitCode=1 Jul 15 14:34:13.319679 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319631 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"61eb6b02f34d84a1eef96d50077432655504eb644a8671078401000088e08724"} Jul 15 14:34:13.319791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"daa917d9fd3ba560e3e2425039caf0400cddd82f4c61e40f897016bada99d338"} Jul 15 14:34:13.319791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"70c91f4e07dec209a936859d498594ed6e402fbebfd684a61c669e109e45178f"} Jul 15 14:34:13.319791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319772 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"ae61cdcdf1305cdfbf77a56fed40d41b54ddef238b433580bca725fb1787937d"} Jul 15 14:34:13.319946 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:13.319801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerDied","Data":"17e106c1c1a63749db7ce8db347d7584d3069c9dcc21fa9a84aa6c8d803817d4"} Jul 15 14:34:14.158556 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.158522 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:14.158795 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:14.158634 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:14.326291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.326257 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5f8v" event={"ID":"792ca10f-5da9-4995-a253-91b8609a1585","Type":"ContainerStarted","Data":"72d469220df3a2960bccfcc603a2c35fb9ad48cc6c972c7e02dd94ff8a725993"} Jul 15 14:34:14.330784 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.330763 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 14:34:14.345524 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.345469 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w5f8v" podStartSLOduration=5.508784843 podStartE2EDuration="24.345448802s" totalImagesPullingTime="18.836663959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:14.343632498 +0000 UTC m=+24.753251777" watchObservedRunningTime="2026-07-15 14:34:14.345448802 +0000 UTC m=+24.755068149" Jul 15 14:34:14.950145 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.949878 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:34:14.950759 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:14.950685 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:34:15.038289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.038139 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T14:34:14.330778949Z","UUID":"1ee09cec-c8f6-4cc4-9410-0f7fa521d360","Handler":null,"Name":"","Endpoint":""} Jul 15 14:34:15.040339 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.040306 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 14:34:15.040473 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.040352 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 14:34:15.159616 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.159084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:15.159616 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:15.159210 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:15.159616 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.159603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:15.159923 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:15.159707 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:15.331651 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.331623 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:34:15.332136 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.332012 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"22c5b0525fc1c0f7b2f8f4aa57fd2538057d89ffa9d57cfd370c152f74c5f3ba"} Jul 15 14:34:15.334000 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:15.333965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" event={"ID":"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd","Type":"ContainerStarted","Data":"773d455e7acc14c15fcb28a32aac70f48534a5a787a19c4dc8e11e91c6c29e8e"} Jul 15 14:34:16.159214 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:16.159170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:16.159398 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:16.159313 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:16.338640 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:16.338605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" event={"ID":"8c5e33d9-83cd-41df-80a4-f606b2e1e8fd","Type":"ContainerStarted","Data":"3e8b9280cf65df22ba0f16b8131231bb27e7f3017375373b52b4f9527ece4120"} Jul 15 14:34:16.361693 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:16.361631 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phxmb" podStartSLOduration=4.300651822 podStartE2EDuration="26.361605208s" totalImagesPullingTime="22.060953386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:16.360641089 +0000 UTC m=+26.770260368" watchObservedRunningTime="2026-07-15 14:34:16.361605208 +0000 UTC m=+26.771224494" Jul 15 14:34:17.158306 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:17.158250 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:17.158306 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:17.158288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:17.158554 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:17.158392 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:17.158618 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:17.158564 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:18.121948 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.121702 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:34:18.122637 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.122039 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 14:34:18.122637 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.122455 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p67nb" Jul 15 14:34:18.158899 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.158866 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:18.159039 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:18.158977 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:18.344436 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.344401 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="fd0a78768411963c8cd7c13566396d84407b50242105f44a7f3893ee15840697" exitCode=0 Jul 15 14:34:18.344619 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.344490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"fd0a78768411963c8cd7c13566396d84407b50242105f44a7f3893ee15840697"} Jul 15 14:34:18.347956 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.347940 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:34:18.348360 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.348337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"f32a2b544d16b4b5cd94cf1381b10227e425350a253790fd6bb6abfa7a82a923"} Jul 15 14:34:18.348782 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:18.348769 2567 scope.go:117] "RemoveContainer" containerID="17e106c1c1a63749db7ce8db347d7584d3069c9dcc21fa9a84aa6c8d803817d4" Jul 15 14:34:19.158488 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.158456 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:19.159047 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.158462 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:19.159047 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:19.158580 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:19.159047 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:19.158632 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:19.355033 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.355011 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:34:19.355460 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.355428 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" event={"ID":"f3f1225f-4a90-4e01-847d-2fe17310bada","Type":"ContainerStarted","Data":"e8cf7f732ee80da056f4888417eaa12c3f46ae177c5547335def532d6cd24e58"} Jul 15 14:34:19.355777 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.355513 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 14:34:19.362802 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.362776 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:19.362946 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.362864 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:19.374599 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.374567 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:19.380895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.380829 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:19.439162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.438898 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" podStartSLOduration=10.512337722 podStartE2EDuration="29.438879474s" totalImagesPullingTime="18.926541752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:19.399053143 +0000 UTC m=+29.808672417" watchObservedRunningTime="2026-07-15 14:34:19.438879474 +0000 UTC m=+29.848498750" Jul 15 14:34:19.616731 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.616700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-864x5"] Jul 15 14:34:19.616917 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.616834 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:19.616982 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:19.616928 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:19.621103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.620932 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rwcbt"] Jul 15 14:34:19.621103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.621081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:19.621302 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:19.621209 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:19.621770 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.621743 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9z5t2"] Jul 15 14:34:19.621899 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:19.621885 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:19.622016 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:19.621998 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:20.248157 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:20.248120 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:20.360164 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:20.360124 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="57443b95acb24e18c3a490850513ab38458202d4d599884d90ffb148303fb097" exitCode=0 Jul 15 14:34:20.360326 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:20.360205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"57443b95acb24e18c3a490850513ab38458202d4d599884d90ffb148303fb097"} Jul 15 14:34:21.159202 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:21.159168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:21.159382 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:21.159168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:21.159382 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:21.159287 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:21.159382 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:21.159168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:21.159382 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:21.159360 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:21.159583 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:21.159436 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:22.367937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:22.367894 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="86f824aa76cb0b4b7ec81ac85bccf9a0084584ca22a06d651741a1d222d77f59" exitCode=0 Jul 15 14:34:22.368532 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:22.367986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"86f824aa76cb0b4b7ec81ac85bccf9a0084584ca22a06d651741a1d222d77f59"} Jul 15 14:34:22.993380 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:22.993355 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77j4x_716f201f-0bc0-4fb7-92a4-124c180171bd/dns-node-resolver/0.log" Jul 15 14:34:23.158450 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.158421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:23.158624 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.158417 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:23.158624 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.158571 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9z5t2" podUID="a736a3ad-c62f-45ec-974e-2902d5c4ee62" Jul 15 14:34:23.158624 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.158614 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:23.158759 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.158722 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-864x5" podUID="2a835c6a-ccd5-40db-b075-a1437d022fc1" Jul 15 14:34:23.158856 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.158822 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwcbt" podUID="40583480-876e-4dc7-81f2-e5d037961c1a" Jul 15 14:34:23.736952 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.736861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:23.736952 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.736920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:23.737417 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.737019 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:23.737417 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.737022 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:23.737417 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.737109 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret podName:2a835c6a-ccd5-40db-b075-a1437d022fc1 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:55.737088156 +0000 UTC m=+66.146707428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret") pod "global-pull-secret-syncer-864x5" (UID: "2a835c6a-ccd5-40db-b075-a1437d022fc1") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:23.737417 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.737128 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs podName:a736a3ad-c62f-45ec-974e-2902d5c4ee62 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:55.737118391 +0000 UTC m=+66.146737653 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs") pod "network-metrics-daemon-9z5t2" (UID: "a736a3ad-c62f-45ec-974e-2902d5c4ee62") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:23.938757 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.938560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:23.938903 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.938732 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:34:23.938903 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.938839 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:34:23.938903 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.938852 2567 projected.go:196] Error preparing data for projected volume kube-api-access-xq8h5 for pod openshift-network-diagnostics/network-check-target-rwcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:23.939032 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:23.938910 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5 podName:40583480-876e-4dc7-81f2-e5d037961c1a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:55.938892754 +0000 UTC m=+66.348512010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq8h5" (UniqueName: "kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5") pod "network-check-target-rwcbt" (UID: "40583480-876e-4dc7-81f2-e5d037961c1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:23.960966 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:23.960938 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nzwnl_d13942e3-ce89-4455-9a17-7681469756ce/node-ca/0.log" Jul 15 14:34:24.949031 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:24.948902 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-12.ec2.internal" event="NodeReady" Jul 15 14:34:24.949476 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:24.949084 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 14:34:25.055825 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.055795 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9bjkb"] Jul 15 14:34:25.059044 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.059017 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m5dj6"] Jul 15 14:34:25.059194 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.059148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.060826 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.060806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.066872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.066542 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 14:34:25.066872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.066603 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 14:34:25.066872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.066654 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 14:34:25.066872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.066741 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5q8j8\"" Jul 15 14:34:25.067176 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.067049 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 14:34:25.069475 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.069449 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 14:34:25.069938 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.069897 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 14:34:25.070034 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.070014 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 14:34:25.071049 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.071031 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8nlk\"" Jul 15 14:34:25.079013 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.078989 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9bjkb"] Jul 15 14:34:25.100495 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.100460 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bd88j"] Jul 15 14:34:25.102335 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.102317 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.117034 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.117005 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5dj6"] Jul 15 14:34:25.118472 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.118449 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 14:34:25.118760 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.118739 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w66fn\"" Jul 15 14:34:25.121011 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.120990 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bd88j"] Jul 15 14:34:25.122270 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.122250 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 14:34:25.148492 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj94q\" (UniqueName: \"kubernetes.io/projected/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-api-access-fj94q\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.148656 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148497 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c338dfc-d036-4d2a-96cb-9e35564de89e-data-volume\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.148656 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.148656 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148592 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d249ec3-554d-48f7-b365-e22dc3f2d482-cert\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.148656 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.148830 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c338dfc-d036-4d2a-96cb-9e35564de89e-crio-socket\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.148830 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.148764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78qw\" (UniqueName: \"kubernetes.io/projected/3d249ec3-554d-48f7-b365-e22dc3f2d482-kube-api-access-m78qw\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.158358 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.158318 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:25.158531 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.158317 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:25.158531 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.158329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:25.164909 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.164887 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 14:34:25.165050 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.164936 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 14:34:25.165306 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.165287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tz86z\"" Jul 15 14:34:25.165510 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.165373 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m26tp\"" Jul 15 14:34:25.165889 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.165865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 14:34:25.189976 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.189942 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 14:34:25.249667 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.249667 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d249ec3-554d-48f7-b365-e22dc3f2d482-cert\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.249667 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/926b471b-8345-4e7a-828f-931d85a39e04-config-volume\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c338dfc-d036-4d2a-96cb-9e35564de89e-crio-socket\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.249749 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.249790 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249755 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl68f\" (UniqueName: \"kubernetes.io/projected/926b471b-8345-4e7a-828f-931d85a39e04-kube-api-access-jl68f\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.249840 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.749815852 +0000 UTC m=+36.159435109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.249858 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.749849843 +0000 UTC m=+36.159469099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : configmap "kube-rbac-proxy" not found Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78qw\" (UniqueName: \"kubernetes.io/projected/3d249ec3-554d-48f7-b365-e22dc3f2d482-kube-api-access-m78qw\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.249940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c338dfc-d036-4d2a-96cb-9e35564de89e-crio-socket\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.250241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj94q\" (UniqueName: \"kubernetes.io/projected/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-api-access-fj94q\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.250241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.249993 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.250241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.250017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926b471b-8345-4e7a-828f-931d85a39e04-tmp-dir\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.250241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.250049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c338dfc-d036-4d2a-96cb-9e35564de89e-data-volume\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.250396 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.250375 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c338dfc-d036-4d2a-96cb-9e35564de89e-data-volume\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.254303 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.254280 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d249ec3-554d-48f7-b365-e22dc3f2d482-cert\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.283231 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.283203 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78qw\" (UniqueName: \"kubernetes.io/projected/3d249ec3-554d-48f7-b365-e22dc3f2d482-kube-api-access-m78qw\") pod \"ingress-canary-m5dj6\" (UID: \"3d249ec3-554d-48f7-b365-e22dc3f2d482\") " pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.283384 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.283203 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj94q\" (UniqueName: \"kubernetes.io/projected/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-api-access-fj94q\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.320493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.320459 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-nspj9"] Jul 15 14:34:25.323618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.323594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.349632 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.349601 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 14:34:25.350384 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350360 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 14:34:25.350508 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350372 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g54ts\"" Jul 15 14:34:25.350508 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.350601 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926b471b-8345-4e7a-828f-931d85a39e04-tmp-dir\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.350601 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/926b471b-8345-4e7a-828f-931d85a39e04-config-volume\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.350699 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.350605 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:25.350699 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl68f\" (UniqueName: \"kubernetes.io/projected/926b471b-8345-4e7a-828f-931d85a39e04-kube-api-access-jl68f\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.350699 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.350670 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls podName:926b471b-8345-4e7a-828f-931d85a39e04 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.850648638 +0000 UTC m=+36.260267911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls") pod "dns-default-bd88j" (UID: "926b471b-8345-4e7a-828f-931d85a39e04") : secret "dns-default-metrics-tls" not found Jul 15 14:34:25.350859 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.350839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926b471b-8345-4e7a-828f-931d85a39e04-tmp-dir\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.351074 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.351042 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 14:34:25.352110 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.352091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/926b471b-8345-4e7a-828f-931d85a39e04-config-volume\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.353929 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.353905 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 14:34:25.368900 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.368867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-nspj9"] Jul 15 14:34:25.377146 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.377114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5dj6" Jul 15 14:34:25.387024 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.386997 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl68f\" (UniqueName: \"kubernetes.io/projected/926b471b-8345-4e7a-828f-931d85a39e04-kube-api-access-jl68f\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.454391 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.454289 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-cabundle\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.454884 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.454853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-key\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.454986 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.454893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnjn\" (UniqueName: \"kubernetes.io/projected/803f3bba-63da-4689-9553-0ff4d00bec6a-kube-api-access-9pnjn\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.556089 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.556039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-cabundle\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.556285 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.556178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-key\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.556285 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.556210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnjn\" (UniqueName: \"kubernetes.io/projected/803f3bba-63da-4689-9553-0ff4d00bec6a-kube-api-access-9pnjn\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.556843 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.556819 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-cabundle\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.558850 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.558825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/803f3bba-63da-4689-9553-0ff4d00bec6a-signing-key\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.568536 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.568515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnjn\" (UniqueName: \"kubernetes.io/projected/803f3bba-63da-4689-9553-0ff4d00bec6a-kube-api-access-9pnjn\") pod \"service-ca-79f68b9f4f-nspj9\" (UID: \"803f3bba-63da-4689-9553-0ff4d00bec6a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.633515 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.633476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" Jul 15 14:34:25.758095 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.758041 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.758257 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.758121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:25.758257 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.758228 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.758257 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.758244 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:25.758379 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.758308 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:26.758287963 +0000 UTC m=+37.167907236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.758379 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.758329 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:26.758318932 +0000 UTC m=+37.167938194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : configmap "kube-rbac-proxy" not found Jul 15 14:34:25.858923 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:25.858841 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:25.859107 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.858968 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:25.859107 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:25.859034 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls podName:926b471b-8345-4e7a-828f-931d85a39e04 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:26.85901491 +0000 UTC m=+37.268634166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls") pod "dns-default-bd88j" (UID: "926b471b-8345-4e7a-828f-931d85a39e04") : secret "dns-default-metrics-tls" not found Jul 15 14:34:26.362263 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.362230 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5dj6"] Jul 15 14:34:26.370473 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.370439 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-nspj9"] Jul 15 14:34:26.373672 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:26.373644 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d249ec3_554d_48f7_b365_e22dc3f2d482.slice/crio-b7efe29ba2fd90a76349a72b795c438a2b72a9a6c80c0894ff3e8cf6a77375c9 WatchSource:0}: Error finding container b7efe29ba2fd90a76349a72b795c438a2b72a9a6c80c0894ff3e8cf6a77375c9: Status 404 returned error can't find the container with id b7efe29ba2fd90a76349a72b795c438a2b72a9a6c80c0894ff3e8cf6a77375c9 Jul 15 14:34:26.373887 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:26.373866 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803f3bba_63da_4689_9553_0ff4d00bec6a.slice/crio-d4eb374d13f960953b28d920ee80d9b576f973b25611aa62f2404ddfc6771fd8 WatchSource:0}: Error finding container d4eb374d13f960953b28d920ee80d9b576f973b25611aa62f2404ddfc6771fd8: Status 404 returned error can't find the container with id d4eb374d13f960953b28d920ee80d9b576f973b25611aa62f2404ddfc6771fd8 Jul 15 14:34:26.382809 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.382785 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" event={"ID":"803f3bba-63da-4689-9553-0ff4d00bec6a","Type":"ContainerStarted","Data":"d4eb374d13f960953b28d920ee80d9b576f973b25611aa62f2404ddfc6771fd8"} Jul 15 14:34:26.383942 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.383906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5dj6" event={"ID":"3d249ec3-554d-48f7-b365-e22dc3f2d482","Type":"ContainerStarted","Data":"b7efe29ba2fd90a76349a72b795c438a2b72a9a6c80c0894ff3e8cf6a77375c9"} Jul 15 14:34:26.386761 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.386740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerStarted","Data":"a84eee5446ba39f469c0a0af7554673c730bdfc3e53d355935dced68a45d4959"} Jul 15 14:34:26.767765 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.767671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:26.767765 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.767712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:26.767957 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.767790 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:26.767957 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.767825 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:26.767957 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.767842 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:28.767827058 +0000 UTC m=+39.177446315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : configmap "kube-rbac-proxy" not found Jul 15 14:34:26.767957 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.767877 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:28.767864054 +0000 UTC m=+39.177483311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:26.868595 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:26.868563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:26.868753 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.868692 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:26.868808 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:26.868757 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls podName:926b471b-8345-4e7a-828f-931d85a39e04 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:28.868739236 +0000 UTC m=+39.278358508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls") pod "dns-default-bd88j" (UID: "926b471b-8345-4e7a-828f-931d85a39e04") : secret "dns-default-metrics-tls" not found Jul 15 14:34:27.393289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:27.393247 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="a84eee5446ba39f469c0a0af7554673c730bdfc3e53d355935dced68a45d4959" exitCode=0 Jul 15 14:34:27.393739 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:27.393330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"a84eee5446ba39f469c0a0af7554673c730bdfc3e53d355935dced68a45d4959"} Jul 15 14:34:28.399732 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:28.399697 2567 generic.go:356] "Generic (PLEG): container finished" podID="62e20b9a-ec6b-4119-8892-4666e75413fb" containerID="b158560408726090c5bbebcb87a9c854adfa64bb35f397a82b8935a034efec42" exitCode=0 Jul 15 14:34:28.400275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:28.399764 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerDied","Data":"b158560408726090c5bbebcb87a9c854adfa64bb35f397a82b8935a034efec42"} Jul 15 14:34:28.783005 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:28.782922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:28.783005 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:28.782985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:28.783217 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.783105 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:28.783217 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.783112 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:28.783217 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.783174 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:32.783152536 +0000 UTC m=+43.192771807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : configmap "kube-rbac-proxy" not found Jul 15 14:34:28.783217 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.783192 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:32.78318263 +0000 UTC m=+43.192801898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:28.884199 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:28.884154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:28.884351 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.884327 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:28.884427 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:28.884415 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls podName:926b471b-8345-4e7a-828f-931d85a39e04 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:32.884393948 +0000 UTC m=+43.294013230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls") pod "dns-default-bd88j" (UID: "926b471b-8345-4e7a-828f-931d85a39e04") : secret "dns-default-metrics-tls" not found Jul 15 14:34:30.406546 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.406255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" event={"ID":"803f3bba-63da-4689-9553-0ff4d00bec6a","Type":"ContainerStarted","Data":"1a596765e1b1cda0a52b462659fa28dd797cb125d0c5f89ddec13605a9609d6b"} Jul 15 14:34:30.407672 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.407647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5dj6" event={"ID":"3d249ec3-554d-48f7-b365-e22dc3f2d482","Type":"ContainerStarted","Data":"4f2356f1c27d74c4140daa6c57c7f5141bc6c51dd406f74f6a6cf2ff89a4aeb8"} Jul 15 14:34:30.411116 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.411093 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9474" event={"ID":"62e20b9a-ec6b-4119-8892-4666e75413fb","Type":"ContainerStarted","Data":"babb0964fca4ba3033023335e8360c890192962ef7ba46beb2dbde09a15c66d2"} Jul 15 14:34:30.428329 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.428279 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-nspj9" podStartSLOduration=1.768425857 podStartE2EDuration="5.428263342s" totalImagesPullingTime="3.659837485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:30.425405944 +0000 UTC m=+40.835025222" watchObservedRunningTime="2026-07-15 14:34:30.428263342 +0000 UTC m=+40.837882599" Jul 15 14:34:30.448766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.448685 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m5dj6" podStartSLOduration=1.7830184249999999 podStartE2EDuration="5.448670092s" totalImagesPullingTime="3.665651667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:30.447896416 +0000 UTC m=+40.857515693" watchObservedRunningTime="2026-07-15 14:34:30.448670092 +0000 UTC m=+40.858289371" Jul 15 14:34:30.494970 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:30.494918 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t9474" podStartSLOduration=11.438986753 podStartE2EDuration="40.494901147s" totalImagesPullingTime="29.055914394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:30.492687893 +0000 UTC m=+40.902307172" watchObservedRunningTime="2026-07-15 14:34:30.494901147 +0000 UTC m=+40.904520424" Jul 15 14:34:32.814536 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:32.814492 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:32.814896 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:32.814584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:32.814896 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:32.814682 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:32.814896 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:32.814734 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls podName:0c338dfc-d036-4d2a-96cb-9e35564de89e nodeName:}" failed. No retries permitted until 2026-07-15 14:34:40.814721108 +0000 UTC m=+51.224340364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bjkb" (UID: "0c338dfc-d036-4d2a-96cb-9e35564de89e") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:32.815111 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:32.815092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c338dfc-d036-4d2a-96cb-9e35564de89e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:32.915082 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:32.915027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:32.915238 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:32.915178 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:32.915309 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:32.915244 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls podName:926b471b-8345-4e7a-828f-931d85a39e04 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:40.915227727 +0000 UTC m=+51.324846984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls") pod "dns-default-bd88j" (UID: "926b471b-8345-4e7a-828f-931d85a39e04") : secret "dns-default-metrics-tls" not found Jul 15 14:34:40.872870 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:40.872832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:40.876918 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:40.876896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c338dfc-d036-4d2a-96cb-9e35564de89e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bjkb\" (UID: \"0c338dfc-d036-4d2a-96cb-9e35564de89e\") " pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:40.971581 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:40.971548 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9bjkb" Jul 15 14:34:40.973798 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:40.973778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:40.975879 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:40.975858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/926b471b-8345-4e7a-828f-931d85a39e04-metrics-tls\") pod \"dns-default-bd88j\" (UID: \"926b471b-8345-4e7a-828f-931d85a39e04\") " pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:41.012115 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:41.012083 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:41.123835 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:41.123809 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9bjkb"] Jul 15 14:34:41.127088 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:41.127030 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c338dfc_d036_4d2a_96cb_9e35564de89e.slice/crio-798831432b39c0766bbe21fce9d5d91f671a59773f3e6ce156aef31a7116d9b7 WatchSource:0}: Error finding container 798831432b39c0766bbe21fce9d5d91f671a59773f3e6ce156aef31a7116d9b7: Status 404 returned error can't find the container with id 798831432b39c0766bbe21fce9d5d91f671a59773f3e6ce156aef31a7116d9b7 Jul 15 14:34:41.163232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:41.163208 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bd88j"] Jul 15 14:34:41.164581 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:41.164554 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926b471b_8345_4e7a_828f_931d85a39e04.slice/crio-2e719f31f23a9dad92b9fd358ff32b69ae64d35d19397e456110c472c3cbf497 WatchSource:0}: Error finding container 2e719f31f23a9dad92b9fd358ff32b69ae64d35d19397e456110c472c3cbf497: Status 404 returned error can't find the container with id 2e719f31f23a9dad92b9fd358ff32b69ae64d35d19397e456110c472c3cbf497 Jul 15 14:34:41.439281 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:41.439191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bjkb" event={"ID":"0c338dfc-d036-4d2a-96cb-9e35564de89e","Type":"ContainerStarted","Data":"798831432b39c0766bbe21fce9d5d91f671a59773f3e6ce156aef31a7116d9b7"} Jul 15 14:34:41.440267 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:41.440244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd88j" event={"ID":"926b471b-8345-4e7a-828f-931d85a39e04","Type":"ContainerStarted","Data":"2e719f31f23a9dad92b9fd358ff32b69ae64d35d19397e456110c472c3cbf497"} Jul 15 14:34:47.459788 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:47.459743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd88j" event={"ID":"926b471b-8345-4e7a-828f-931d85a39e04","Type":"ContainerStarted","Data":"d8e57431351a11216e930d3ed715674d59b4e8975fa13629a8d3c0942174a60c"} Jul 15 14:34:47.460323 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:47.459813 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:47.460323 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:47.459835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd88j" event={"ID":"926b471b-8345-4e7a-828f-931d85a39e04","Type":"ContainerStarted","Data":"66ed7bd993440f2a9d807c2847d219881fa3c5f7ee27ef82669aed5db7bbd434"} Jul 15 14:34:47.461950 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:47.461916 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bjkb" event={"ID":"0c338dfc-d036-4d2a-96cb-9e35564de89e","Type":"ContainerStarted","Data":"b78c556d287ff75a3358cf07fc7f1ff151b7438feefe555b3af37df07f715d8a"} Jul 15 14:34:48.839052 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.838997 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bd88j" podStartSLOduration=18.555445541 podStartE2EDuration="23.838977515s" totalImagesPullingTime="5.283531974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:47.481568832 +0000 UTC m=+57.891188114" watchObservedRunningTime="2026-07-15 14:34:48.838977515 +0000 UTC m=+59.248596794" Jul 15 14:34:48.839763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.839744 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g"] Jul 15 14:34:48.852444 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.852247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:48.860463 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.860096 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 14:34:48.862899 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.862870 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 14:34:48.862899 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.862899 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm"] Jul 15 14:34:48.866094 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.865840 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g"] Jul 15 14:34:48.866094 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.865950 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:48.868702 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.868676 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 14:34:48.868818 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.868775 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rh4b5\"" Jul 15 14:34:48.876280 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.876251 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 14:34:48.878694 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.878662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 14:34:48.900948 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.900908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm"] Jul 15 14:34:48.943371 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.943330 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7d6j\" (UniqueName: \"kubernetes.io/projected/f1cf4aa4-4994-433d-9a15-ef7be8a34196-kube-api-access-p7d6j\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:48.943371 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.943374 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/77e8e58b-d646-482b-b262-777f62e707d1-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:48.943574 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.943425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cll\" (UniqueName: \"kubernetes.io/projected/77e8e58b-d646-482b-b262-777f62e707d1-kube-api-access-w4cll\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:48.943574 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.943450 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1cf4aa4-4994-433d-9a15-ef7be8a34196-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:48.943653 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:48.943590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77e8e58b-d646-482b-b262-777f62e707d1-tmp\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.044719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.044687 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7d6j\" (UniqueName: \"kubernetes.io/projected/f1cf4aa4-4994-433d-9a15-ef7be8a34196-kube-api-access-p7d6j\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:49.044719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.044725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/77e8e58b-d646-482b-b262-777f62e707d1-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.044963 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.044780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cll\" (UniqueName: \"kubernetes.io/projected/77e8e58b-d646-482b-b262-777f62e707d1-kube-api-access-w4cll\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.044963 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.044814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1cf4aa4-4994-433d-9a15-ef7be8a34196-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:49.044963 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.044885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77e8e58b-d646-482b-b262-777f62e707d1-tmp\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.045348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.045324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77e8e58b-d646-482b-b262-777f62e707d1-tmp\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.047857 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.047828 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/77e8e58b-d646-482b-b262-777f62e707d1-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.047857 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.047848 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1cf4aa4-4994-433d-9a15-ef7be8a34196-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:49.121199 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.121129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7d6j\" (UniqueName: \"kubernetes.io/projected/f1cf4aa4-4994-433d-9a15-ef7be8a34196-kube-api-access-p7d6j\") pod \"managed-serviceaccount-addon-agent-544b9979d4-gwb2g\" (UID: \"f1cf4aa4-4994-433d-9a15-ef7be8a34196\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:49.146495 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.146447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cll\" (UniqueName: \"kubernetes.io/projected/77e8e58b-d646-482b-b262-777f62e707d1-kube-api-access-w4cll\") pod \"klusterlet-addon-workmgr-56f84b96bc-mzkdm\" (UID: \"77e8e58b-d646-482b-b262-777f62e707d1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.166072 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.166016 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bfbb8fbb9-pwttq"] Jul 15 14:34:49.174229 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.174190 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.175284 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.175262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" Jul 15 14:34:49.177794 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.177541 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 14:34:49.178167 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.178113 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2mcw\"" Jul 15 14:34:49.178265 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.178183 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 14:34:49.178553 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.178536 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 14:34:49.184990 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.184971 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:49.189411 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.189390 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 14:34:49.196919 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.196887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfbb8fbb9-pwttq"] Jul 15 14:34:49.246473 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-certificates\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246488 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-bound-sa-token\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-trusted-ca\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64n7\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-kube-api-access-l64n7\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-installation-pull-secrets\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-tls\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc93a353-0e25-49a9-b35c-0800dcc8aeff-ca-trust-extracted\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.246751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.246738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-image-registry-private-configuration\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347756 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-trusted-ca\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347949 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64n7\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-kube-api-access-l64n7\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347949 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-installation-pull-secrets\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347949 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-tls\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347949 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc93a353-0e25-49a9-b35c-0800dcc8aeff-ca-trust-extracted\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.347949 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-image-registry-private-configuration\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.348270 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.347970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-certificates\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.348270 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.348011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-bound-sa-token\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.348905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.348734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc93a353-0e25-49a9-b35c-0800dcc8aeff-ca-trust-extracted\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.348905 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.348839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-trusted-ca\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.349112 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.349078 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-certificates\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.350829 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.350801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-installation-pull-secrets\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.350922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.350801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc93a353-0e25-49a9-b35c-0800dcc8aeff-image-registry-private-configuration\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.351074 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.351031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-registry-tls\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.367447 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.367396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64n7\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-kube-api-access-l64n7\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.368411 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.368386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc93a353-0e25-49a9-b35c-0800dcc8aeff-bound-sa-token\") pod \"image-registry-5bfbb8fbb9-pwttq\" (UID: \"bc93a353-0e25-49a9-b35c-0800dcc8aeff\") " pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:49.485164 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:49.485074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:50.173518 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.173488 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g"] Jul 15 14:34:50.179836 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:50.177279 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1cf4aa4_4994_433d_9a15_ef7be8a34196.slice/crio-e2e76615844527e1f380404f50ec6c3df1562609ff219a23f78ca6547846ad98 WatchSource:0}: Error finding container e2e76615844527e1f380404f50ec6c3df1562609ff219a23f78ca6547846ad98: Status 404 returned error can't find the container with id e2e76615844527e1f380404f50ec6c3df1562609ff219a23f78ca6547846ad98 Jul 15 14:34:50.283289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.283262 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfbb8fbb9-pwttq"] Jul 15 14:34:50.285585 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:50.285536 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc93a353_0e25_49a9_b35c_0800dcc8aeff.slice/crio-6affe4008429112f835187c79f11053e76ebe8a1a01827d72f2a84a9b69aaa81 WatchSource:0}: Error finding container 6affe4008429112f835187c79f11053e76ebe8a1a01827d72f2a84a9b69aaa81: Status 404 returned error can't find the container with id 6affe4008429112f835187c79f11053e76ebe8a1a01827d72f2a84a9b69aaa81 Jul 15 14:34:50.302629 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.302605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm"] Jul 15 14:34:50.305754 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:50.305729 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e8e58b_d646_482b_b262_777f62e707d1.slice/crio-f68d9f09a13e5e0c2bc99473dba0f007ff1a7532d5c4ceef55ed47990c3c17c5 WatchSource:0}: Error finding container f68d9f09a13e5e0c2bc99473dba0f007ff1a7532d5c4ceef55ed47990c3c17c5: Status 404 returned error can't find the container with id f68d9f09a13e5e0c2bc99473dba0f007ff1a7532d5c4ceef55ed47990c3c17c5 Jul 15 14:34:50.472872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.472837 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bjkb" event={"ID":"0c338dfc-d036-4d2a-96cb-9e35564de89e","Type":"ContainerStarted","Data":"a0af4604cd927762446676c853a9f1bdd3031fa37c6b977658136fd1c0b7694f"} Jul 15 14:34:50.474357 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.474331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" event={"ID":"bc93a353-0e25-49a9-b35c-0800dcc8aeff","Type":"ContainerStarted","Data":"49f75d4d72dfe216ad2a2f3d0a0f41faf6857d8f643b470bdea093e550ca2c90"} Jul 15 14:34:50.474497 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.474376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" event={"ID":"bc93a353-0e25-49a9-b35c-0800dcc8aeff","Type":"ContainerStarted","Data":"6affe4008429112f835187c79f11053e76ebe8a1a01827d72f2a84a9b69aaa81"} Jul 15 14:34:50.474497 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.474421 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:34:50.475531 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.475510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" event={"ID":"f1cf4aa4-4994-433d-9a15-ef7be8a34196","Type":"ContainerStarted","Data":"e2e76615844527e1f380404f50ec6c3df1562609ff219a23f78ca6547846ad98"} Jul 15 14:34:50.476526 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.476504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" event={"ID":"77e8e58b-d646-482b-b262-777f62e707d1","Type":"ContainerStarted","Data":"f68d9f09a13e5e0c2bc99473dba0f007ff1a7532d5c4ceef55ed47990c3c17c5"} Jul 15 14:34:50.500850 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:50.500763 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" podStartSLOduration=1.5007436379999999 podStartE2EDuration="1.500743638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:50.499826631 +0000 UTC m=+60.909445908" watchObservedRunningTime="2026-07-15 14:34:50.500743638 +0000 UTC m=+60.910362918" Jul 15 14:34:51.384877 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:51.383488 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v25jf" Jul 15 14:34:52.517739 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.516700 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9"] Jul 15 14:34:52.523839 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.523124 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:52.526866 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.526224 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9jrl6\"" Jul 15 14:34:52.526866 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.526496 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 14:34:52.535479 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.535446 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9"] Jul 15 14:34:52.577960 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.577827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bmmf9\" (UID: \"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:52.678710 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:52.678557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bmmf9\" (UID: \"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:52.678887 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:52.678741 2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 15 14:34:52.678887 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:34:52.678819 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates podName:a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:53.178797529 +0000 UTC m=+63.588416804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-bmmf9" (UID: "a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35") : secret "prometheus-operator-admission-webhook-tls" not found Jul 15 14:34:53.183093 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:53.183035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bmmf9\" (UID: \"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:53.185968 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:53.185939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bmmf9\" (UID: \"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:53.438373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:53.438285 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:54.037386 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.037350 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:34:54.039634 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.039615 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.043665 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.043629 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 14:34:54.044002 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.043974 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 14:34:54.044002 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.043974 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 14:34:54.044461 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.044431 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 14:34:54.044572 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.044499 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 14:34:54.044572 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.044507 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 14:34:54.044700 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.044579 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 14:34:54.045304 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.045240 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tkzjz\"" Jul 15 14:34:54.057396 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.057370 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:34:54.092634 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.092822 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9j8\" (UniqueName: \"kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.092822 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092787 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.092935 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.092935 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.093022 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.092935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.193886 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.193848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.193914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.193942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.193966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9j8\" (UniqueName: \"kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.194044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.194101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.194672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194826 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.194672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.194962 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.194935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.196816 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.196792 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.197043 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.197018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.204138 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.204114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9j8\" (UniqueName: \"kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8\") pod \"console-f57fdb495-hd26t\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:54.350705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:54.350660 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:34:55.808871 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.808830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:55.809359 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.808942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:55.811513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.811490 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 14:34:55.812148 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.812127 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 14:34:55.822322 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.822290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a835c6a-ccd5-40db-b075-a1437d022fc1-original-pull-secret\") pod \"global-pull-secret-syncer-864x5\" (UID: \"2a835c6a-ccd5-40db-b075-a1437d022fc1\") " pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:55.822468 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:55.822297 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736a3ad-c62f-45ec-974e-2902d5c4ee62-metrics-certs\") pod \"network-metrics-daemon-9z5t2\" (UID: \"a736a3ad-c62f-45ec-974e-2902d5c4ee62\") " pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:56.011521 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.011476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:56.014588 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.014559 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 14:34:56.024562 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.024532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 14:34:56.035424 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.035387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq8h5\" (UniqueName: \"kubernetes.io/projected/40583480-876e-4dc7-81f2-e5d037961c1a-kube-api-access-xq8h5\") pod \"network-check-target-rwcbt\" (UID: \"40583480-876e-4dc7-81f2-e5d037961c1a\") " pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:56.071378 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.071294 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tz86z\"" Jul 15 14:34:56.077394 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.077357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-864x5" Jul 15 14:34:56.079393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.079365 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9z5t2" Jul 15 14:34:56.084919 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.084893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m26tp\"" Jul 15 14:34:56.092935 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.092905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:34:56.515645 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.515538 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:56.521992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.517029 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" podUID="77e8e58b-d646-482b-b262-777f62e707d1" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.11:8000/readyz\": dial tcp 10.134.0.11:8000: connect: connection refused" Jul 15 14:34:56.577800 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.576494 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" podStartSLOduration=2.503845333 podStartE2EDuration="8.576474202s" totalImagesPullingTime="6.072628869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:56.576113813 +0000 UTC m=+66.985733092" watchObservedRunningTime="2026-07-15 14:34:56.576474202 +0000 UTC m=+66.986093483" Jul 15 14:34:56.608359 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.608260 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9z5t2"] Jul 15 14:34:56.612653 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.612583 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9"] Jul 15 14:34:56.613872 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:56.613823 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda736a3ad_c62f_45ec_974e_2902d5c4ee62.slice/crio-9b3e1d4e051625804642dd06d1608200da196c13e85e89ccbb5732f0b42f0b6f WatchSource:0}: Error finding container 9b3e1d4e051625804642dd06d1608200da196c13e85e89ccbb5732f0b42f0b6f: Status 404 returned error can't find the container with id 9b3e1d4e051625804642dd06d1608200da196c13e85e89ccbb5732f0b42f0b6f Jul 15 14:34:56.617565 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:56.617533 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ae71fb_8de0_45c4_a2e9_e3ab9d7acc35.slice/crio-41c5fee89156948aef336929f4e3a0a7601dcb843969ff6910f153eb3bc6f8ad WatchSource:0}: Error finding container 41c5fee89156948aef336929f4e3a0a7601dcb843969ff6910f153eb3bc6f8ad: Status 404 returned error can't find the container with id 41c5fee89156948aef336929f4e3a0a7601dcb843969ff6910f153eb3bc6f8ad Jul 15 14:34:56.649442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.649415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-864x5"] Jul 15 14:34:56.651332 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.651311 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rwcbt"] Jul 15 14:34:56.653374 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:56.653345 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a835c6a_ccd5_40db_b075_a1437d022fc1.slice/crio-90df70301182e9f23a3b9ad8c837d27d4065bface0dd703311456a1f2befbf00 WatchSource:0}: Error finding container 90df70301182e9f23a3b9ad8c837d27d4065bface0dd703311456a1f2befbf00: Status 404 returned error can't find the container with id 90df70301182e9f23a3b9ad8c837d27d4065bface0dd703311456a1f2befbf00 Jul 15 14:34:56.659151 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:56.659034 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40583480_876e_4dc7_81f2_e5d037961c1a.slice/crio-fc71eddbdd9e6ba002b13591fa49d78832a9380388b275b3ce9c4446c39d7551 WatchSource:0}: Error finding container fc71eddbdd9e6ba002b13591fa49d78832a9380388b275b3ce9c4446c39d7551: Status 404 returned error can't find the container with id fc71eddbdd9e6ba002b13591fa49d78832a9380388b275b3ce9c4446c39d7551 Jul 15 14:34:56.666453 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:56.666431 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:34:56.669817 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:34:56.669791 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3863b9c8_846d_4c39_9586_2959b242f9a5.slice/crio-34348e08dfb2692a3033d8196563f895cc73af23579646c7cb0dfbc7984ec2aa WatchSource:0}: Error finding container 34348e08dfb2692a3033d8196563f895cc73af23579646c7cb0dfbc7984ec2aa: Status 404 returned error can't find the container with id 34348e08dfb2692a3033d8196563f895cc73af23579646c7cb0dfbc7984ec2aa Jul 15 14:34:57.468262 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.468172 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bd88j" Jul 15 14:34:57.522305 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.522247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" event={"ID":"f1cf4aa4-4994-433d-9a15-ef7be8a34196","Type":"ContainerStarted","Data":"865ec304ea285236269285030fd2f8d768e790eb3eff529c2a686ac885eae964"} Jul 15 14:34:57.524073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.523964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" event={"ID":"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35","Type":"ContainerStarted","Data":"41c5fee89156948aef336929f4e3a0a7601dcb843969ff6910f153eb3bc6f8ad"} Jul 15 14:34:57.526490 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.526433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9z5t2" event={"ID":"a736a3ad-c62f-45ec-974e-2902d5c4ee62","Type":"ContainerStarted","Data":"9b3e1d4e051625804642dd06d1608200da196c13e85e89ccbb5732f0b42f0b6f"} Jul 15 14:34:57.528130 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.528084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f57fdb495-hd26t" event={"ID":"3863b9c8-846d-4c39-9586-2959b242f9a5","Type":"ContainerStarted","Data":"34348e08dfb2692a3033d8196563f895cc73af23579646c7cb0dfbc7984ec2aa"} Jul 15 14:34:57.529910 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.529878 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" event={"ID":"77e8e58b-d646-482b-b262-777f62e707d1","Type":"ContainerStarted","Data":"87bacbc8abd7c4e92c48b2eaa0febf0e48819095cf4e30d04d5c6048a339c458"} Jul 15 14:34:57.531726 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.531662 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f84b96bc-mzkdm" Jul 15 14:34:57.532230 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.532204 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rwcbt" event={"ID":"40583480-876e-4dc7-81f2-e5d037961c1a","Type":"ContainerStarted","Data":"fc71eddbdd9e6ba002b13591fa49d78832a9380388b275b3ce9c4446c39d7551"} Jul 15 14:34:57.534867 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.534845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bjkb" event={"ID":"0c338dfc-d036-4d2a-96cb-9e35564de89e","Type":"ContainerStarted","Data":"159b6a6671effc0b818df4237a882dfbd83997e1aff2dab643b2307a6d57c004"} Jul 15 14:34:57.536219 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.536197 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-864x5" event={"ID":"2a835c6a-ccd5-40db-b075-a1437d022fc1","Type":"ContainerStarted","Data":"90df70301182e9f23a3b9ad8c837d27d4065bface0dd703311456a1f2befbf00"} Jul 15 14:34:57.549786 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.549747 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-544b9979d4-gwb2g" podStartSLOduration=3.3621291380000002 podStartE2EDuration="9.549732074s" totalImagesPullingTime="6.187602936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:57.547438726 +0000 UTC m=+67.957058004" watchObservedRunningTime="2026-07-15 14:34:57.549732074 +0000 UTC m=+67.959351352" Jul 15 14:34:57.611234 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:57.611079 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9bjkb" podStartSLOduration=17.483215901 podStartE2EDuration="32.611043948s" totalImagesPullingTime="15.127828047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:34:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:57.609582502 +0000 UTC m=+68.019201780" watchObservedRunningTime="2026-07-15 14:34:57.611043948 +0000 UTC m=+68.020663208" Jul 15 14:34:58.541553 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:58.541449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" event={"ID":"a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35","Type":"ContainerStarted","Data":"afdb4dc4b278418953a0701295312e97be51ef69ade3007aab6ff096b511d5e7"} Jul 15 14:34:58.542195 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:58.542147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:58.545317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:58.545219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9z5t2" event={"ID":"a736a3ad-c62f-45ec-974e-2902d5c4ee62","Type":"ContainerStarted","Data":"6479f5bdcc6f2884d8de6295e5ffa3e021e2d9e805bede033f6a5d32adc08eb4"} Jul 15 14:34:58.547707 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:58.547672 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" Jul 15 14:34:58.564181 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:58.564087 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bmmf9" podStartSLOduration=4.850202038 podStartE2EDuration="6.564027229s" totalImagesPullingTime="1.713825191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:58.562235098 +0000 UTC m=+68.971854379" watchObservedRunningTime="2026-07-15 14:34:58.564027229 +0000 UTC m=+68.973646520" Jul 15 14:34:59.553392 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:59.553353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9z5t2" event={"ID":"a736a3ad-c62f-45ec-974e-2902d5c4ee62","Type":"ContainerStarted","Data":"bea86780f198b3711f124ca0a2225eebd8f7efdb8cab2b5bafd7b351fcab2c2f"} Jul 15 14:34:59.584101 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:34:59.583327 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9z5t2" podStartSLOduration=67.868487683 podStartE2EDuration="1m9.583307267s" totalImagesPullingTime="1.714819584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:59.582282916 +0000 UTC m=+69.991902194" watchObservedRunningTime="2026-07-15 14:34:59.583307267 +0000 UTC m=+69.992926545" Jul 15 14:35:00.558751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:00.558365 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rwcbt" event={"ID":"40583480-876e-4dc7-81f2-e5d037961c1a","Type":"ContainerStarted","Data":"183c102f5acdd5a5ae5f8594d5d8039fbcac16fc2c884123dca719ef9dfb17cf"} Jul 15 14:35:00.558751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:00.558527 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:35:00.578438 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:00.578388 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rwcbt" podStartSLOduration=67.161327757 podStartE2EDuration="1m10.578368623s" totalImagesPullingTime="3.417040866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:00.577617971 +0000 UTC m=+70.987237249" watchObservedRunningTime="2026-07-15 14:35:00.578368623 +0000 UTC m=+70.987987902" Jul 15 14:35:01.399052 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.399015 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:01.426582 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.426554 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:01.426740 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.426663 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.434831 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.434804 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 14:35:01.550007 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.549981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550098 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sck\" (UniqueName: \"kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.550419 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.550193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.650934 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.650864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.650934 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.650907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.650967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48sck\" (UniqueName: \"kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.650992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651518 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651784 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651972 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651932 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.651972 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.651932 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.653999 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.653977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.654256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.654240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.661932 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.661911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sck\" (UniqueName: \"kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck\") pod \"console-68b495d744-x5hdz\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:01.739925 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:01.739893 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:03.118838 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.118808 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:03.123895 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:03.123867 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef6ea3f_e238_4ed9_8600_cab30e6b821c.slice/crio-5adcfad786dc902609f8461911af0c51cb25f0c653513f43472f8ad9253c0f2d WatchSource:0}: Error finding container 5adcfad786dc902609f8461911af0c51cb25f0c653513f43472f8ad9253c0f2d: Status 404 returned error can't find the container with id 5adcfad786dc902609f8461911af0c51cb25f0c653513f43472f8ad9253c0f2d Jul 15 14:35:03.572442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.572399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-864x5" event={"ID":"2a835c6a-ccd5-40db-b075-a1437d022fc1","Type":"ContainerStarted","Data":"65b6650de6624ba3f513297a1b60800677f077b0416d04e53384add23d47a407"} Jul 15 14:35:03.574209 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.574178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f57fdb495-hd26t" event={"ID":"3863b9c8-846d-4c39-9586-2959b242f9a5","Type":"ContainerStarted","Data":"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc"} Jul 15 14:35:03.575793 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.575769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b495d744-x5hdz" event={"ID":"def6ea3f-e238-4ed9-8600-cab30e6b821c","Type":"ContainerStarted","Data":"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675"} Jul 15 14:35:03.575897 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.575811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b495d744-x5hdz" event={"ID":"def6ea3f-e238-4ed9-8600-cab30e6b821c","Type":"ContainerStarted","Data":"5adcfad786dc902609f8461911af0c51cb25f0c653513f43472f8ad9253c0f2d"} Jul 15 14:35:03.592310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.592263 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-864x5" podStartSLOduration=67.269335297 podStartE2EDuration="1m13.592244582s" totalImagesPullingTime="6.322909285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:03.590734077 +0000 UTC m=+74.000353355" watchObservedRunningTime="2026-07-15 14:35:03.592244582 +0000 UTC m=+74.001863839" Jul 15 14:35:03.611615 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.611571 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f57fdb495-hd26t" podStartSLOduration=3.302876758 podStartE2EDuration="9.611557751s" totalImagesPullingTime="6.308680993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:03.611136853 +0000 UTC m=+74.020756132" watchObservedRunningTime="2026-07-15 14:35:03.611557751 +0000 UTC m=+74.021177028" Jul 15 14:35:03.634763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:03.634727 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68b495d744-x5hdz" podStartSLOduration=2.634715479 podStartE2EDuration="2.634715479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:03.63427254 +0000 UTC m=+74.043891818" watchObservedRunningTime="2026-07-15 14:35:03.634715479 +0000 UTC m=+74.044334756" Jul 15 14:35:04.350768 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:04.350744 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:04.351086 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:04.350789 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:04.355385 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:04.355368 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:04.584624 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:04.584600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:05.109684 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.109651 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5kd7p"] Jul 15 14:35:05.112983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.112963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.115445 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.115337 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 14:35:05.115943 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.115921 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 14:35:05.116537 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.116515 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 14:35:05.117788 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.117563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4jhc4\"" Jul 15 14:35:05.117788 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.117651 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 14:35:05.121654 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.118007 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 14:35:05.121654 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.118283 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 14:35:05.127202 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.126593 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5kd7p"] Jul 15 14:35:05.141948 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.141925 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x6zrw"] Jul 15 14:35:05.144560 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.144543 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.148300 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.148283 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cdwhh\"" Jul 15 14:35:05.148459 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.148443 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 14:35:05.148547 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.148476 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 14:35:05.148724 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.148709 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 14:35:05.177268 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.177395 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.177395 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.177395 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn58c\" (UniqueName: \"kubernetes.io/projected/fc89f221-3793-40f7-83dc-bd0adf685698-kube-api-access-cn58c\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.177543 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.177543 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.177461 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc89f221-3793-40f7-83dc-bd0adf685698-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.278533 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.278674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn58c\" (UniqueName: \"kubernetes.io/projected/fc89f221-3793-40f7-83dc-bd0adf685698-kube-api-access-cn58c\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.278674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4slh\" (UniqueName: \"kubernetes.io/projected/455d8a67-95db-4f62-8224-00ef2e4208ad-kube-api-access-k4slh\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.278674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278645 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-textfile\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-root\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-tls\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-sys\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc89f221-3793-40f7-83dc-bd0adf685698-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-wtmp\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.278874 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278887 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-metrics-client-ca\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.278936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:05.278948 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:05.279007 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls podName:fc89f221-3793-40f7-83dc-bd0adf685698 nodeName:}" failed. No retries permitted until 2026-07-15 14:35:05.778988253 +0000 UTC m=+76.188607527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-5kd7p" (UID: "fc89f221-3793-40f7-83dc-bd0adf685698") : secret "kube-state-metrics-tls" not found Jul 15 14:35:05.279232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.279192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc89f221-3793-40f7-83dc-bd0adf685698-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.279536 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.279254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.279591 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.279557 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc89f221-3793-40f7-83dc-bd0adf685698-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.281079 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.281038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.295640 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.295619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn58c\" (UniqueName: \"kubernetes.io/projected/fc89f221-3793-40f7-83dc-bd0adf685698-kube-api-access-cn58c\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.380115 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4slh\" (UniqueName: \"kubernetes.io/projected/455d8a67-95db-4f62-8224-00ef2e4208ad-kube-api-access-k4slh\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380115 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-textfile\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380129 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-root\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-tls\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-sys\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-wtmp\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-metrics-client-ca\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-root\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-sys\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-wtmp\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-textfile\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380964 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.380964 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.380799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455d8a67-95db-4f62-8224-00ef2e4208ad-metrics-client-ca\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.382385 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.382368 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-tls\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.382459 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.382440 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/455d8a67-95db-4f62-8224-00ef2e4208ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.390046 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.390025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4slh\" (UniqueName: \"kubernetes.io/projected/455d8a67-95db-4f62-8224-00ef2e4208ad-kube-api-access-k4slh\") pod \"node-exporter-x6zrw\" (UID: \"455d8a67-95db-4f62-8224-00ef2e4208ad\") " pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.455460 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.455441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x6zrw" Jul 15 14:35:05.465434 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:05.465415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455d8a67_95db_4f62_8224_00ef2e4208ad.slice/crio-593864307d768d79968704cf4f956a1591f44ce9b0cc7fb3ff649e927fcab140 WatchSource:0}: Error finding container 593864307d768d79968704cf4f956a1591f44ce9b0cc7fb3ff649e927fcab140: Status 404 returned error can't find the container with id 593864307d768d79968704cf4f956a1591f44ce9b0cc7fb3ff649e927fcab140 Jul 15 14:35:05.586239 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.586215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6zrw" event={"ID":"455d8a67-95db-4f62-8224-00ef2e4208ad","Type":"ContainerStarted","Data":"593864307d768d79968704cf4f956a1591f44ce9b0cc7fb3ff649e927fcab140"} Jul 15 14:35:05.783685 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.783629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:05.785684 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:05.785666 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc89f221-3793-40f7-83dc-bd0adf685698-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5kd7p\" (UID: \"fc89f221-3793-40f7-83dc-bd0adf685698\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:06.030518 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:06.030497 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" Jul 15 14:35:06.163826 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:06.163801 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5kd7p"] Jul 15 14:35:06.167606 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:06.166746 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc89f221_3793_40f7_83dc_bd0adf685698.slice/crio-0f0729631cc053c5219a32a309d665aba09271eee3d5519351e7dd9f55464185 WatchSource:0}: Error finding container 0f0729631cc053c5219a32a309d665aba09271eee3d5519351e7dd9f55464185: Status 404 returned error can't find the container with id 0f0729631cc053c5219a32a309d665aba09271eee3d5519351e7dd9f55464185 Jul 15 14:35:06.590719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:06.590686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" event={"ID":"fc89f221-3793-40f7-83dc-bd0adf685698","Type":"ContainerStarted","Data":"0f0729631cc053c5219a32a309d665aba09271eee3d5519351e7dd9f55464185"} Jul 15 14:35:07.597773 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:07.597737 2567 generic.go:356] "Generic (PLEG): container finished" podID="455d8a67-95db-4f62-8224-00ef2e4208ad" containerID="06683dfd4b92afed2eb6fc893a8cc976630a6c99e577214eace0aca69120bb06" exitCode=0 Jul 15 14:35:07.598234 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:07.597826 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6zrw" event={"ID":"455d8a67-95db-4f62-8224-00ef2e4208ad","Type":"ContainerDied","Data":"06683dfd4b92afed2eb6fc893a8cc976630a6c99e577214eace0aca69120bb06"} Jul 15 14:35:08.604723 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:08.604688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" event={"ID":"fc89f221-3793-40f7-83dc-bd0adf685698","Type":"ContainerStarted","Data":"4285474cdcb639f58b5dd6fc489a59703df6a9947abc8da0142d98f5ccf7bf47"} Jul 15 14:35:08.605132 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:08.604763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" event={"ID":"fc89f221-3793-40f7-83dc-bd0adf685698","Type":"ContainerStarted","Data":"cd83439f580b035da543ea9165981055189a6074fe41ebc513652a37c34f4b95"} Jul 15 14:35:08.607311 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:08.607281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6zrw" event={"ID":"455d8a67-95db-4f62-8224-00ef2e4208ad","Type":"ContainerStarted","Data":"46aa3fac8e610731c232f199e2cdd2eb4c3661e94d09b7c54d7aca8adce38b73"} Jul 15 14:35:08.607395 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:08.607339 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6zrw" event={"ID":"455d8a67-95db-4f62-8224-00ef2e4208ad","Type":"ContainerStarted","Data":"131f15f553e1b1893ebfb3cd132f1a5e7d171b448afa86d0a5ca25489613cc6b"} Jul 15 14:35:08.633070 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:08.633002 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x6zrw" podStartSLOduration=1.986295843 podStartE2EDuration="3.632983251s" totalImagesPullingTime="1.585464364s" totalInitContainerRuntime="61.223044ms" isStatefulPod=true podCreationTimestamp="2026-07-15 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:08.631789986 +0000 UTC m=+79.041409266" watchObservedRunningTime="2026-07-15 14:35:08.632983251 +0000 UTC m=+79.042602530" Jul 15 14:35:09.613682 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.613648 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" event={"ID":"fc89f221-3793-40f7-83dc-bd0adf685698","Type":"ContainerStarted","Data":"e8dfee4c83717828d915a2cc6026f27f73e6127b186219d215ec1147cc044645"} Jul 15 14:35:09.639742 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.639691 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-5kd7p" podStartSLOduration=2.3452999 podStartE2EDuration="4.639676936s" totalImagesPullingTime="2.294377036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:09.637951866 +0000 UTC m=+80.047571135" watchObservedRunningTime="2026-07-15 14:35:09.639676936 +0000 UTC m=+80.049296213" Jul 15 14:35:09.652558 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.652533 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f67b9cd4-xnmnp"] Jul 15 14:35:09.655505 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.655488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.657589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.657570 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 14:35:09.657840 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.657802 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dd8mttomdc07a\"" Jul 15 14:35:09.658193 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.658172 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4kz9h\"" Jul 15 14:35:09.658256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.658197 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 14:35:09.658256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.658220 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 14:35:09.658369 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.658197 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 14:35:09.667163 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.667142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f67b9cd4-xnmnp"] Jul 15 14:35:09.818803 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-client-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.818937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818807 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/778fd302-84b5-46be-8c74-9a249efbfd80-audit-log\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.818937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pc9\" (UniqueName: \"kubernetes.io/projected/778fd302-84b5-46be-8c74-9a249efbfd80-kube-api-access-x2pc9\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.818937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.818937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-tls\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.818937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-metrics-server-audit-profiles\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.819125 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.818951 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-client-certs\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.842038 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.842017 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg"] Jul 15 14:35:09.852168 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.852149 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:09.854678 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.854642 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 14:35:09.854678 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.854664 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bkwlk\"" Jul 15 14:35:09.856918 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.856897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg"] Jul 15 14:35:09.920073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.919996 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-client-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/778fd302-84b5-46be-8c74-9a249efbfd80-audit-log\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920043 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pc9\" (UniqueName: \"kubernetes.io/projected/778fd302-84b5-46be-8c74-9a249efbfd80-kube-api-access-x2pc9\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920293 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920293 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-tls\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920293 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-metrics-server-audit-profiles\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920293 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-client-certs\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920483 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/778fd302-84b5-46be-8c74-9a249efbfd80-audit-log\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.920851 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.920831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.921101 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.921085 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/778fd302-84b5-46be-8c74-9a249efbfd80-metrics-server-audit-profiles\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.922562 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.922541 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-client-ca-bundle\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.922827 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.922806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-tls\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.922888 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.922860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/778fd302-84b5-46be-8c74-9a249efbfd80-secret-metrics-server-client-certs\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.929256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.929234 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pc9\" (UniqueName: \"kubernetes.io/projected/778fd302-84b5-46be-8c74-9a249efbfd80-kube-api-access-x2pc9\") pod \"metrics-server-5f67b9cd4-xnmnp\" (UID: \"778fd302-84b5-46be-8c74-9a249efbfd80\") " pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:09.964695 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:09.964676 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:10.021281 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.021251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.021390 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.021316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgkx\" (UniqueName: \"kubernetes.io/projected/997666de-8abc-471c-a073-02bbf8412f27-kube-api-access-scgkx\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.115941 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.115913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f67b9cd4-xnmnp"] Jul 15 14:35:10.119338 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:10.119310 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778fd302_84b5_46be_8c74_9a249efbfd80.slice/crio-26e4b646e8e8e97d10b1d0d7b7490f53f4646eaaada67649842f507f08d59e02 WatchSource:0}: Error finding container 26e4b646e8e8e97d10b1d0d7b7490f53f4646eaaada67649842f507f08d59e02: Status 404 returned error can't find the container with id 26e4b646e8e8e97d10b1d0d7b7490f53f4646eaaada67649842f507f08d59e02 Jul 15 14:35:10.121839 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.121746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.121839 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.121787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgkx\" (UniqueName: \"kubernetes.io/projected/997666de-8abc-471c-a073-02bbf8412f27-kube-api-access-scgkx\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.121945 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:10.121861 2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 15 14:35:10.121945 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:10.121919 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert podName:997666de-8abc-471c-a073-02bbf8412f27 nodeName:}" failed. No retries permitted until 2026-07-15 14:35:10.62190138 +0000 UTC m=+81.031520639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-csnwg" (UID: "997666de-8abc-471c-a073-02bbf8412f27") : secret "monitoring-plugin-cert" not found Jul 15 14:35:10.134789 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.134767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgkx\" (UniqueName: \"kubernetes.io/projected/997666de-8abc-471c-a073-02bbf8412f27-kube-api-access-scgkx\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.617931 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.617868 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" event={"ID":"778fd302-84b5-46be-8c74-9a249efbfd80","Type":"ContainerStarted","Data":"26e4b646e8e8e97d10b1d0d7b7490f53f4646eaaada67649842f507f08d59e02"} Jul 15 14:35:10.625507 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.625486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.627712 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.627688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/997666de-8abc-471c-a073-02bbf8412f27-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-csnwg\" (UID: \"997666de-8abc-471c-a073-02bbf8412f27\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.629122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.629103 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:10.671966 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.671945 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:35:10.674250 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.674232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.689952 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.689935 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:35:10.761892 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.761870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:10.827189 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827312 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827312 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827259 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827312 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827470 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827470 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.827470 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.827441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhj6\" (UniqueName: \"kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.891782 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.891756 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg"] Jul 15 14:35:10.893839 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:10.893816 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997666de_8abc_471c_a073_02bbf8412f27.slice/crio-fc14a527d5337401ea28ace6d466ae42891114ba5ee2f183da21a30a371157f2 WatchSource:0}: Error finding container fc14a527d5337401ea28ace6d466ae42891114ba5ee2f183da21a30a371157f2: Status 404 returned error can't find the container with id fc14a527d5337401ea28ace6d466ae42891114ba5ee2f183da21a30a371157f2 Jul 15 14:35:10.928031 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928145 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928145 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928145 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhj6\" (UniqueName: \"kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.928289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.928251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.929149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.929102 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.929578 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.929554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.929960 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.929915 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.930150 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.930130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.930786 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.930768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.930870 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.930820 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.944965 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.944946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhj6\" (UniqueName: \"kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6\") pod \"console-5679bddc68-25st6\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:10.982448 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:10.982422 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:11.132230 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.132133 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:35:11.134522 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:35:11.134501 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e5bb6d_bf85_4101_b0b3_02aa108a8518.slice/crio-8b007661b881989317064efb128087fe9786a01b5d1ca02c9b13d30601c8151e WatchSource:0}: Error finding container 8b007661b881989317064efb128087fe9786a01b5d1ca02c9b13d30601c8151e: Status 404 returned error can't find the container with id 8b007661b881989317064efb128087fe9786a01b5d1ca02c9b13d30601c8151e Jul 15 14:35:11.494037 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.493964 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bfbb8fbb9-pwttq" Jul 15 14:35:11.627451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.627260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679bddc68-25st6" event={"ID":"38e5bb6d-bf85-4101-b0b3-02aa108a8518","Type":"ContainerStarted","Data":"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817"} Jul 15 14:35:11.627451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.627321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679bddc68-25st6" event={"ID":"38e5bb6d-bf85-4101-b0b3-02aa108a8518","Type":"ContainerStarted","Data":"8b007661b881989317064efb128087fe9786a01b5d1ca02c9b13d30601c8151e"} Jul 15 14:35:11.629240 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.629099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" event={"ID":"997666de-8abc-471c-a073-02bbf8412f27","Type":"ContainerStarted","Data":"fc14a527d5337401ea28ace6d466ae42891114ba5ee2f183da21a30a371157f2"} Jul 15 14:35:11.740697 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:11.740633 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:12.638341 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.638305 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" event={"ID":"778fd302-84b5-46be-8c74-9a249efbfd80","Type":"ContainerStarted","Data":"d5317a207a1ea1d0752207c20ea383806df6b41318e9009700fc22944bee6770"} Jul 15 14:35:12.639856 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.639818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" event={"ID":"997666de-8abc-471c-a073-02bbf8412f27","Type":"ContainerStarted","Data":"a576ea1fcfddaeccf9d7eb6ce8b72056d61b47ca06fe48c721158e061234a0ff"} Jul 15 14:35:12.640151 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.640136 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:12.645073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.645030 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" Jul 15 14:35:12.659641 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.659597 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5679bddc68-25st6" podStartSLOduration=2.6595824070000003 podStartE2EDuration="2.659582407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:11.663781672 +0000 UTC m=+82.073400961" watchObservedRunningTime="2026-07-15 14:35:12.659582407 +0000 UTC m=+83.069201685" Jul 15 14:35:12.660553 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.660514 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" podStartSLOduration=1.672461439 podStartE2EDuration="3.6605031s" totalImagesPullingTime="1.988041661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:12.658032018 +0000 UTC m=+83.067651296" watchObservedRunningTime="2026-07-15 14:35:12.6605031 +0000 UTC m=+83.070122380" Jul 15 14:35:12.685716 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:12.685681 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-csnwg" podStartSLOduration=2.470771921 podStartE2EDuration="3.685669196s" totalImagesPullingTime="1.214897275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:12.684825626 +0000 UTC m=+83.094444907" watchObservedRunningTime="2026-07-15 14:35:12.685669196 +0000 UTC m=+83.095288485" Jul 15 14:35:20.982751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:20.982718 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:20.983299 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:20.982778 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:20.988223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:20.988194 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:21.677388 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:21.677355 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:35:21.742129 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:21.742096 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:35:29.964828 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:29.964799 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:29.965252 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:29.964901 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:31.564154 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:31.564131 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rwcbt" Jul 15 14:35:35.648101 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.647995 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68b495d744-x5hdz" podUID="def6ea3f-e238-4ed9-8600-cab30e6b821c" containerName="console" containerID="cri-o://3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675" gracePeriod=15 Jul 15 14:35:35.878620 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.878597 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:35.905723 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.905656 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.905723 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.905693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48sck\" (UniqueName: \"kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.905896 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.905737 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.905977 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.905961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.906033 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.905988 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.906033 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906021 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.906154 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906050 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca\") pod \"def6ea3f-e238-4ed9-8600-cab30e6b821c\" (UID: \"def6ea3f-e238-4ed9-8600-cab30e6b821c\") " Jul 15 14:35:35.906435 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906307 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:35.906435 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906365 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:35.906612 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906442 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config" (OuterVolumeSpecName: "console-config") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:35.906612 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.906543 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca" (OuterVolumeSpecName: "service-ca") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:35.908045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.908007 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:35.908263 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.908187 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck" (OuterVolumeSpecName: "kube-api-access-48sck") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "kube-api-access-48sck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:35:35.908412 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:35.908391 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "def6ea3f-e238-4ed9-8600-cab30e6b821c" (UID: "def6ea3f-e238-4ed9-8600-cab30e6b821c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:36.007003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.006975 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-oauth-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.006998 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-trusted-ca-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.007007 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.007016 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.007024 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/def6ea3f-e238-4ed9-8600-cab30e6b821c-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.007032 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/def6ea3f-e238-4ed9-8600-cab30e6b821c-console-oauth-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.007190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.007039 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48sck\" (UniqueName: \"kubernetes.io/projected/def6ea3f-e238-4ed9-8600-cab30e6b821c-kube-api-access-48sck\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:36.726844 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.726803 2567 generic.go:356] "Generic (PLEG): container finished" podID="def6ea3f-e238-4ed9-8600-cab30e6b821c" containerID="3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675" exitCode=0 Jul 15 14:35:36.727320 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.726873 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b495d744-x5hdz" Jul 15 14:35:36.727320 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.726890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b495d744-x5hdz" event={"ID":"def6ea3f-e238-4ed9-8600-cab30e6b821c","Type":"ContainerDied","Data":"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675"} Jul 15 14:35:36.727320 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.726968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b495d744-x5hdz" event={"ID":"def6ea3f-e238-4ed9-8600-cab30e6b821c","Type":"ContainerDied","Data":"5adcfad786dc902609f8461911af0c51cb25f0c653513f43472f8ad9253c0f2d"} Jul 15 14:35:36.727320 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.726990 2567 scope.go:117] "RemoveContainer" containerID="3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675" Jul 15 14:35:36.734581 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.734563 2567 scope.go:117] "RemoveContainer" containerID="3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675" Jul 15 14:35:36.734825 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:36.734808 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675\": container with ID starting with 3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675 not found: ID does not exist" containerID="3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675" Jul 15 14:35:36.734883 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.734832 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675"} err="failed to get container status \"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675\": rpc error: code = NotFound desc = could not find container \"3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675\": container with ID starting with 3318dc59a34fa928b36395cdb36196e53937782d3a19edb5b80f90cb97ac4675 not found: ID does not exist" Jul 15 14:35:36.751319 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.751295 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:36.756836 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:36.756815 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68b495d744-x5hdz"] Jul 15 14:35:38.163277 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:38.163246 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def6ea3f-e238-4ed9-8600-cab30e6b821c" path="/var/lib/kubelet/pods/def6ea3f-e238-4ed9-8600-cab30e6b821c/volumes" Jul 15 14:35:46.762284 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:46.762232 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f57fdb495-hd26t" podUID="3863b9c8-846d-4c39-9586-2959b242f9a5" containerName="console" containerID="cri-o://7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc" gracePeriod=15 Jul 15 14:35:47.020263 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.020208 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:47.099334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099302 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099449 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099379 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099449 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099405 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099449 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099423 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099592 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099456 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv9j8\" (UniqueName: \"kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099592 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099478 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config\") pod \"3863b9c8-846d-4c39-9586-2959b242f9a5\" (UID: \"3863b9c8-846d-4c39-9586-2959b242f9a5\") " Jul 15 14:35:47.099817 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:47.099817 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099806 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca" (OuterVolumeSpecName: "service-ca") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:47.099909 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.099842 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config" (OuterVolumeSpecName: "console-config") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:47.101398 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.101372 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:47.101530 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.101513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:47.101571 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.101526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8" (OuterVolumeSpecName: "kube-api-access-qv9j8") pod "3863b9c8-846d-4c39-9586-2959b242f9a5" (UID: "3863b9c8-846d-4c39-9586-2959b242f9a5"). InnerVolumeSpecName "kube-api-access-qv9j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:35:47.200461 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200439 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-oauth-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.200461 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200460 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.200579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200469 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-oauth-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.200579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200477 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9j8\" (UniqueName: \"kubernetes.io/projected/3863b9c8-846d-4c39-9586-2959b242f9a5-kube-api-access-qv9j8\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.200579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200485 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3863b9c8-846d-4c39-9586-2959b242f9a5-console-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.200579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.200492 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3863b9c8-846d-4c39-9586-2959b242f9a5-console-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:35:47.771835 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.771807 2567 generic.go:356] "Generic (PLEG): container finished" podID="3863b9c8-846d-4c39-9586-2959b242f9a5" containerID="7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc" exitCode=0 Jul 15 14:35:47.772197 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.771856 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f57fdb495-hd26t" Jul 15 14:35:47.772197 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.771894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f57fdb495-hd26t" event={"ID":"3863b9c8-846d-4c39-9586-2959b242f9a5","Type":"ContainerDied","Data":"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc"} Jul 15 14:35:47.772197 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.771946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f57fdb495-hd26t" event={"ID":"3863b9c8-846d-4c39-9586-2959b242f9a5","Type":"ContainerDied","Data":"34348e08dfb2692a3033d8196563f895cc73af23579646c7cb0dfbc7984ec2aa"} Jul 15 14:35:47.772197 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.771969 2567 scope.go:117] "RemoveContainer" containerID="7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc" Jul 15 14:35:47.780832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.780796 2567 scope.go:117] "RemoveContainer" containerID="7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc" Jul 15 14:35:47.781124 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:35:47.781104 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc\": container with ID starting with 7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc not found: ID does not exist" containerID="7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc" Jul 15 14:35:47.781166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.781135 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc"} err="failed to get container status \"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc\": rpc error: code = NotFound desc = could not find container \"7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc\": container with ID starting with 7360f0f3f6bc23e248b143b7489523e9a20935fb195218e175183ea4d825abcc not found: ID does not exist" Jul 15 14:35:47.794496 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.794470 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:35:47.801129 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:47.801110 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f57fdb495-hd26t"] Jul 15 14:35:48.163642 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:48.163616 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3863b9c8-846d-4c39-9586-2959b242f9a5" path="/var/lib/kubelet/pods/3863b9c8-846d-4c39-9586-2959b242f9a5/volumes" Jul 15 14:35:49.970515 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:49.970484 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:35:49.974476 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:35:49.974447 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f67b9cd4-xnmnp" Jul 15 14:36:33.727423 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727393 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727639 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3863b9c8-846d-4c39-9586-2959b242f9a5" containerName="console" Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727650 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3863b9c8-846d-4c39-9586-2959b242f9a5" containerName="console" Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727664 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def6ea3f-e238-4ed9-8600-cab30e6b821c" containerName="console" Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727670 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="def6ea3f-e238-4ed9-8600-cab30e6b821c" containerName="console" Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727708 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="def6ea3f-e238-4ed9-8600-cab30e6b821c" containerName="console" Jul 15 14:36:33.727832 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.727716 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3863b9c8-846d-4c39-9586-2959b242f9a5" containerName="console" Jul 15 14:36:33.729865 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.729847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.747002 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.746971 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:36:33.840964 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.840923 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.840964 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.840964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.841200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.841022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.841200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.841041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.841200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.841110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.841200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.841136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k628k\" (UniqueName: \"kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.841200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.841188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942319 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942470 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.942707 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.942504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k628k\" (UniqueName: \"kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.943302 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.943275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.943412 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.943330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.943412 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.943330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.943538 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.943515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.944760 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.944743 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.944875 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.944856 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:33.950940 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:33.950919 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k628k\" (UniqueName: \"kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k\") pod \"console-599dcddfbf-92msl\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:34.038741 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:34.038645 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:34.171602 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:34.171579 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:36:34.173501 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:36:34.173477 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f0aa43_b24e_452a_832f_0c8b078cffb1.slice/crio-9b8c77fa47fd54ebd56ae8036ade3267b3bdfae51edc7853da54cf4f96d4ebc4 WatchSource:0}: Error finding container 9b8c77fa47fd54ebd56ae8036ade3267b3bdfae51edc7853da54cf4f96d4ebc4: Status 404 returned error can't find the container with id 9b8c77fa47fd54ebd56ae8036ade3267b3bdfae51edc7853da54cf4f96d4ebc4 Jul 15 14:36:34.938290 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:34.938254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599dcddfbf-92msl" event={"ID":"51f0aa43-b24e-452a-832f-0c8b078cffb1","Type":"ContainerStarted","Data":"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04"} Jul 15 14:36:34.938747 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:34.938302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599dcddfbf-92msl" event={"ID":"51f0aa43-b24e-452a-832f-0c8b078cffb1","Type":"ContainerStarted","Data":"9b8c77fa47fd54ebd56ae8036ade3267b3bdfae51edc7853da54cf4f96d4ebc4"} Jul 15 14:36:34.960275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:34.960220 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599dcddfbf-92msl" podStartSLOduration=1.96020163 podStartE2EDuration="1.96020163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:36:34.958676486 +0000 UTC m=+165.368295764" watchObservedRunningTime="2026-07-15 14:36:34.96020163 +0000 UTC m=+165.369820908" Jul 15 14:36:44.039755 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:44.039662 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:44.039755 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:44.039725 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:44.044648 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:44.044620 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:44.976294 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:44.976265 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:36:45.051644 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:36:45.051606 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:37:10.072327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.072261 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5679bddc68-25st6" podUID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" containerName="console" containerID="cri-o://636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817" gracePeriod=15 Jul 15 14:37:10.316901 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.316873 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:37:10.441950 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.441852 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.441950 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.441896 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.441950 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.441913 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.441950 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.441934 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnhj6\" (UniqueName: \"kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.442276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.441984 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.442276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442028 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.442276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442069 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle\") pod \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\" (UID: \"38e5bb6d-bf85-4101-b0b3-02aa108a8518\") " Jul 15 14:37:10.442433 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442297 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca" (OuterVolumeSpecName: "service-ca") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:37:10.442519 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442496 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config" (OuterVolumeSpecName: "console-config") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:37:10.442570 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442523 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:37:10.442619 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.442569 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:37:10.444164 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.444132 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6" (OuterVolumeSpecName: "kube-api-access-gnhj6") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "kube-api-access-gnhj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:37:10.444419 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.444394 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:37:10.444634 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.444613 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38e5bb6d-bf85-4101-b0b3-02aa108a8518" (UID: "38e5bb6d-bf85-4101-b0b3-02aa108a8518"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:37:10.543223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543183 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543216 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-trusted-ca-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543226 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543234 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38e5bb6d-bf85-4101-b0b3-02aa108a8518-console-oauth-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543486 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543243 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543486 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543252 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhj6\" (UniqueName: \"kubernetes.io/projected/38e5bb6d-bf85-4101-b0b3-02aa108a8518-kube-api-access-gnhj6\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:10.543486 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:10.543260 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38e5bb6d-bf85-4101-b0b3-02aa108a8518-oauth-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:37:11.065350 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.065313 2567 generic.go:356] "Generic (PLEG): container finished" podID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" containerID="636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817" exitCode=0 Jul 15 14:37:11.065529 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.065364 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679bddc68-25st6" Jul 15 14:37:11.065529 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.065397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679bddc68-25st6" event={"ID":"38e5bb6d-bf85-4101-b0b3-02aa108a8518","Type":"ContainerDied","Data":"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817"} Jul 15 14:37:11.065529 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.065439 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679bddc68-25st6" event={"ID":"38e5bb6d-bf85-4101-b0b3-02aa108a8518","Type":"ContainerDied","Data":"8b007661b881989317064efb128087fe9786a01b5d1ca02c9b13d30601c8151e"} Jul 15 14:37:11.065529 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.065462 2567 scope.go:117] "RemoveContainer" containerID="636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817" Jul 15 14:37:11.073787 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.073682 2567 scope.go:117] "RemoveContainer" containerID="636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817" Jul 15 14:37:11.073989 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:37:11.073968 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817\": container with ID starting with 636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817 not found: ID does not exist" containerID="636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817" Jul 15 14:37:11.074027 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.073998 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817"} err="failed to get container status \"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817\": rpc error: code = NotFound desc = could not find container \"636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817\": container with ID starting with 636980867a3e9f023509ea3a6feaf286ec82bd469e9afd3afea469cb85b72817 not found: ID does not exist" Jul 15 14:37:11.089232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.089199 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:37:11.091188 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:11.091165 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5679bddc68-25st6"] Jul 15 14:37:12.162745 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:12.162712 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" path="/var/lib/kubelet/pods/38e5bb6d-bf85-4101-b0b3-02aa108a8518/volumes" Jul 15 14:37:44.609796 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.609755 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:37:44.610276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.610002 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" containerName="console" Jul 15 14:37:44.610276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.610013 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" containerName="console" Jul 15 14:37:44.610276 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.610082 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e5bb6d-bf85-4101-b0b3-02aa108a8518" containerName="console" Jul 15 14:37:44.612709 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.612688 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.627651 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.627623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:37:44.708476 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708488 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcp4k\" (UniqueName: \"kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708627 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.708674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.708650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.809895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.809858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.809895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.809900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.809916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.809957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.809990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.810009 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.810027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcp4k\" (UniqueName: \"kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810741 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.810711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810856 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.810714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.810856 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.810793 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.811199 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.811173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.812547 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.812522 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.812621 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.812561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.819785 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.819763 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcp4k\" (UniqueName: \"kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k\") pod \"console-b9cbff45d-4fnh2\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:44.922040 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:44.921942 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:45.062177 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:45.062151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:37:45.064719 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:37:45.064697 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ec1975_5b9b_48bf_8b51_51d1672a797e.slice/crio-16675b983ce29b0fdea94cf518543a64f5bff17bacb0e132ea75a723c02fdaab WatchSource:0}: Error finding container 16675b983ce29b0fdea94cf518543a64f5bff17bacb0e132ea75a723c02fdaab: Status 404 returned error can't find the container with id 16675b983ce29b0fdea94cf518543a64f5bff17bacb0e132ea75a723c02fdaab Jul 15 14:37:45.188593 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:45.188499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cbff45d-4fnh2" event={"ID":"f8ec1975-5b9b-48bf-8b51-51d1672a797e","Type":"ContainerStarted","Data":"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3"} Jul 15 14:37:45.188593 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:45.188553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cbff45d-4fnh2" event={"ID":"f8ec1975-5b9b-48bf-8b51-51d1672a797e","Type":"ContainerStarted","Data":"16675b983ce29b0fdea94cf518543a64f5bff17bacb0e132ea75a723c02fdaab"} Jul 15 14:37:45.211334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:45.211283 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b9cbff45d-4fnh2" podStartSLOduration=1.211266218 podStartE2EDuration="1.211266218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:37:45.209655764 +0000 UTC m=+235.619275049" watchObservedRunningTime="2026-07-15 14:37:45.211266218 +0000 UTC m=+235.620885495" Jul 15 14:37:54.922751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:54.922709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:54.923192 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:54.922764 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:54.927237 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:54.927217 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:55.225677 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:55.225601 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:37:55.280972 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:37:55.280938 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:38:20.300865 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.300824 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599dcddfbf-92msl" podUID="51f0aa43-b24e-452a-832f-0c8b078cffb1" containerName="console" containerID="cri-o://fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04" gracePeriod=15 Jul 15 14:38:20.552423 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.552351 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:38:20.701289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701246 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701293 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701528 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701348 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701528 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701367 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k628k\" (UniqueName: \"kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701528 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701385 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701528 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701409 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701528 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701429 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config\") pod \"51f0aa43-b24e-452a-832f-0c8b078cffb1\" (UID: \"51f0aa43-b24e-452a-832f-0c8b078cffb1\") " Jul 15 14:38:20.701868 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701840 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:38:20.702008 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701977 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config" (OuterVolumeSpecName: "console-config") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:38:20.702098 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701984 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca" (OuterVolumeSpecName: "service-ca") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:38:20.702098 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.701990 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:38:20.703589 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.703559 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:38:20.704042 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.704021 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:38:20.704169 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.704040 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k" (OuterVolumeSpecName: "kube-api-access-k628k") pod "51f0aa43-b24e-452a-832f-0c8b078cffb1" (UID: "51f0aa43-b24e-452a-832f-0c8b078cffb1"). InnerVolumeSpecName "kube-api-access-k628k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:38:20.802317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802281 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-oauth-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802310 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802318 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-oauth-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802326 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-trusted-ca-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802597 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802335 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51f0aa43-b24e-452a-832f-0c8b078cffb1-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802597 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802345 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51f0aa43-b24e-452a-832f-0c8b078cffb1-console-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:20.802597 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:20.802352 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k628k\" (UniqueName: \"kubernetes.io/projected/51f0aa43-b24e-452a-832f-0c8b078cffb1-kube-api-access-k628k\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:38:21.312999 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.312965 2567 generic.go:356] "Generic (PLEG): container finished" podID="51f0aa43-b24e-452a-832f-0c8b078cffb1" containerID="fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04" exitCode=0 Jul 15 14:38:21.313418 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.313034 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599dcddfbf-92msl" Jul 15 14:38:21.313418 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.313046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599dcddfbf-92msl" event={"ID":"51f0aa43-b24e-452a-832f-0c8b078cffb1","Type":"ContainerDied","Data":"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04"} Jul 15 14:38:21.313418 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.313192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599dcddfbf-92msl" event={"ID":"51f0aa43-b24e-452a-832f-0c8b078cffb1","Type":"ContainerDied","Data":"9b8c77fa47fd54ebd56ae8036ade3267b3bdfae51edc7853da54cf4f96d4ebc4"} Jul 15 14:38:21.313418 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.313221 2567 scope.go:117] "RemoveContainer" containerID="fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04" Jul 15 14:38:21.321785 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.321766 2567 scope.go:117] "RemoveContainer" containerID="fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04" Jul 15 14:38:21.322124 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:38:21.322094 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04\": container with ID starting with fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04 not found: ID does not exist" containerID="fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04" Jul 15 14:38:21.322221 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.322134 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04"} err="failed to get container status \"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04\": rpc error: code = NotFound desc = could not find container \"fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04\": container with ID starting with fa64ac6ed1f24fc4df6020b142ef39cb7b05ec05b118fa59d2cbcb4209645b04 not found: ID does not exist" Jul 15 14:38:21.341881 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.341846 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:38:21.345340 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:21.345314 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599dcddfbf-92msl"] Jul 15 14:38:22.163416 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:22.163388 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f0aa43-b24e-452a-832f-0c8b078cffb1" path="/var/lib/kubelet/pods/51f0aa43-b24e-452a-832f-0c8b078cffb1/volumes" Jul 15 14:38:45.407601 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.407560 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7"] Jul 15 14:38:45.408038 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.407916 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f0aa43-b24e-452a-832f-0c8b078cffb1" containerName="console" Jul 15 14:38:45.408038 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.407930 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f0aa43-b24e-452a-832f-0c8b078cffb1" containerName="console" Jul 15 14:38:45.408038 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.407987 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f0aa43-b24e-452a-832f-0c8b078cffb1" containerName="console" Jul 15 14:38:45.410866 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.410842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.412870 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.412844 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4cxzl\"" Jul 15 14:38:45.413406 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.413386 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 14:38:45.413702 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.413683 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 14:38:45.428499 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.428468 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7"] Jul 15 14:38:45.491163 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.491129 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.491379 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.491170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvv6k\" (UniqueName: \"kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.491379 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.491302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.592571 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.592529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.592571 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.592578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.592820 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.592609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvv6k\" (UniqueName: \"kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.592962 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.592940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.593018 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.592966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.602141 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.602108 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvv6k\" (UniqueName: \"kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.719238 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.719149 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:38:45.851339 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:45.851316 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7"] Jul 15 14:38:45.854021 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:38:45.853995 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37881f46_a263_4212_8517_4c53429970b9.slice/crio-0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b WatchSource:0}: Error finding container 0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b: Status 404 returned error can't find the container with id 0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b Jul 15 14:38:46.404638 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:46.404604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" event={"ID":"37881f46-a263-4212-8517-4c53429970b9","Type":"ContainerStarted","Data":"0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b"} Jul 15 14:38:49.981835 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:49.981806 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:38:49.984200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:49.984177 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:38:49.987620 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:49.987601 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 14:38:55.440006 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:55.439966 2567 generic.go:356] "Generic (PLEG): container finished" podID="37881f46-a263-4212-8517-4c53429970b9" containerID="f8dfb45cf11dba2b20a47d98d758f72c689f1b82625ded1d11c068cdb33a8bb7" exitCode=0 Jul 15 14:38:55.440435 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:55.440014 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" event={"ID":"37881f46-a263-4212-8517-4c53429970b9","Type":"ContainerDied","Data":"f8dfb45cf11dba2b20a47d98d758f72c689f1b82625ded1d11c068cdb33a8bb7"} Jul 15 14:38:55.440992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:55.440972 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:38:58.454308 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:58.454275 2567 generic.go:356] "Generic (PLEG): container finished" podID="37881f46-a263-4212-8517-4c53429970b9" containerID="dcfee0ebbe26d4cbe3f3a5763dea4fc9ebcb17e2cbd1e014c208150ba96d184e" exitCode=0 Jul 15 14:38:58.454706 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:38:58.454371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" event={"ID":"37881f46-a263-4212-8517-4c53429970b9","Type":"ContainerDied","Data":"dcfee0ebbe26d4cbe3f3a5763dea4fc9ebcb17e2cbd1e014c208150ba96d184e"} Jul 15 14:39:05.483067 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:05.482955 2567 generic.go:356] "Generic (PLEG): container finished" podID="37881f46-a263-4212-8517-4c53429970b9" containerID="7aa10d365e4166a0dcd9f7635876cc79f1b42dc181e788240d7d0f2a340ee9c2" exitCode=0 Jul 15 14:39:05.483403 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:05.483050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" event={"ID":"37881f46-a263-4212-8517-4c53429970b9","Type":"ContainerDied","Data":"7aa10d365e4166a0dcd9f7635876cc79f1b42dc181e788240d7d0f2a340ee9c2"} Jul 15 14:39:06.609434 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.609408 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:39:06.663504 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.663470 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle\") pod \"37881f46-a263-4212-8517-4c53429970b9\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " Jul 15 14:39:06.663701 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.663518 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvv6k\" (UniqueName: \"kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k\") pod \"37881f46-a263-4212-8517-4c53429970b9\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " Jul 15 14:39:06.663701 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.663548 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util\") pod \"37881f46-a263-4212-8517-4c53429970b9\" (UID: \"37881f46-a263-4212-8517-4c53429970b9\") " Jul 15 14:39:06.664296 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.664268 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle" (OuterVolumeSpecName: "bundle") pod "37881f46-a263-4212-8517-4c53429970b9" (UID: "37881f46-a263-4212-8517-4c53429970b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:06.665821 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.665792 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k" (OuterVolumeSpecName: "kube-api-access-mvv6k") pod "37881f46-a263-4212-8517-4c53429970b9" (UID: "37881f46-a263-4212-8517-4c53429970b9"). InnerVolumeSpecName "kube-api-access-mvv6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:39:06.667955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.667934 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util" (OuterVolumeSpecName: "util") pod "37881f46-a263-4212-8517-4c53429970b9" (UID: "37881f46-a263-4212-8517-4c53429970b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:06.765118 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.764982 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:06.765118 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.765015 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvv6k\" (UniqueName: \"kubernetes.io/projected/37881f46-a263-4212-8517-4c53429970b9-kube-api-access-mvv6k\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:06.765118 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:06.765029 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37881f46-a263-4212-8517-4c53429970b9-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:07.491341 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:07.491317 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" Jul 15 14:39:07.491341 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:07.491317 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h9pw7" event={"ID":"37881f46-a263-4212-8517-4c53429970b9","Type":"ContainerDied","Data":"0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b"} Jul 15 14:39:07.491541 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:07.491360 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dcb1ccf8defe7df159282de1651a0113901c3f09197cc821ed3b49388504f2b" Jul 15 14:39:18.116395 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116357 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7"] Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116756 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="util" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116773 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="util" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116784 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="pull" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116793 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="pull" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116801 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="extract" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116808 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="extract" Jul 15 14:39:18.116983 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.116874 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="37881f46-a263-4212-8517-4c53429970b9" containerName="extract" Jul 15 14:39:18.119096 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.119075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.124140 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.124120 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4cxzl\"" Jul 15 14:39:18.124290 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.124270 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 14:39:18.124640 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.124615 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 14:39:18.133933 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.133905 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7"] Jul 15 14:39:18.254139 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.254092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.254327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.254165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.254327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.254209 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kszk\" (UniqueName: \"kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.355437 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.355407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kszk\" (UniqueName: \"kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.355530 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.355458 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.355530 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.355486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.355918 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.355897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.355956 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.355909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.373397 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.373321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kszk\" (UniqueName: \"kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.427733 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.427684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:18.567265 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:18.567239 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7"] Jul 15 14:39:18.569826 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:39:18.569798 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847902fa_57b5_4332_a138_a46f7623f1da.slice/crio-f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5 WatchSource:0}: Error finding container f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5: Status 404 returned error can't find the container with id f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5 Jul 15 14:39:19.541886 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:19.541850 2567 generic.go:356] "Generic (PLEG): container finished" podID="847902fa-57b5-4332-a138-a46f7623f1da" containerID="36f1f5b490e0e96b0fd151fc02d44ba001956a97bba9a634c2207e17ffb18a72" exitCode=0 Jul 15 14:39:19.542259 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:19.541930 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" event={"ID":"847902fa-57b5-4332-a138-a46f7623f1da","Type":"ContainerDied","Data":"36f1f5b490e0e96b0fd151fc02d44ba001956a97bba9a634c2207e17ffb18a72"} Jul 15 14:39:19.542259 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:19.541978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" event={"ID":"847902fa-57b5-4332-a138-a46f7623f1da","Type":"ContainerStarted","Data":"f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5"} Jul 15 14:39:22.555477 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:22.555446 2567 generic.go:356] "Generic (PLEG): container finished" podID="847902fa-57b5-4332-a138-a46f7623f1da" containerID="9c5c88d4e5f1bf1a4e1812cf7afe7881d66a8df2d0002e20f37ffaed6a000290" exitCode=0 Jul 15 14:39:22.555903 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:22.555511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" event={"ID":"847902fa-57b5-4332-a138-a46f7623f1da","Type":"ContainerDied","Data":"9c5c88d4e5f1bf1a4e1812cf7afe7881d66a8df2d0002e20f37ffaed6a000290"} Jul 15 14:39:23.560775 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:23.560741 2567 generic.go:356] "Generic (PLEG): container finished" podID="847902fa-57b5-4332-a138-a46f7623f1da" containerID="c8bfc19ddfdf50ea9375aec7edb6828673ce07e13859bb42eaf571f98b970213" exitCode=0 Jul 15 14:39:23.561293 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:23.560833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" event={"ID":"847902fa-57b5-4332-a138-a46f7623f1da","Type":"ContainerDied","Data":"c8bfc19ddfdf50ea9375aec7edb6828673ce07e13859bb42eaf571f98b970213"} Jul 15 14:39:24.690212 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.690184 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:24.811742 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.811708 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle\") pod \"847902fa-57b5-4332-a138-a46f7623f1da\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " Jul 15 14:39:24.811965 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.811787 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util\") pod \"847902fa-57b5-4332-a138-a46f7623f1da\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " Jul 15 14:39:24.811965 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.811824 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kszk\" (UniqueName: \"kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk\") pod \"847902fa-57b5-4332-a138-a46f7623f1da\" (UID: \"847902fa-57b5-4332-a138-a46f7623f1da\") " Jul 15 14:39:24.812220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.812186 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle" (OuterVolumeSpecName: "bundle") pod "847902fa-57b5-4332-a138-a46f7623f1da" (UID: "847902fa-57b5-4332-a138-a46f7623f1da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:24.813928 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.813904 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk" (OuterVolumeSpecName: "kube-api-access-2kszk") pod "847902fa-57b5-4332-a138-a46f7623f1da" (UID: "847902fa-57b5-4332-a138-a46f7623f1da"). InnerVolumeSpecName "kube-api-access-2kszk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:39:24.816220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.816157 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util" (OuterVolumeSpecName: "util") pod "847902fa-57b5-4332-a138-a46f7623f1da" (UID: "847902fa-57b5-4332-a138-a46f7623f1da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:24.913003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.912956 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:24.913003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.912998 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kszk\" (UniqueName: \"kubernetes.io/projected/847902fa-57b5-4332-a138-a46f7623f1da-kube-api-access-2kszk\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:24.913003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:24.913010 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/847902fa-57b5-4332-a138-a46f7623f1da-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:25.570034 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:25.570008 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" Jul 15 14:39:25.570034 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:25.570017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnldg7" event={"ID":"847902fa-57b5-4332-a138-a46f7623f1da","Type":"ContainerDied","Data":"f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5"} Jul 15 14:39:25.570264 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:25.570071 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f52e2b90969efed443cd82b9b6d5f8840e90bdf324701d2fa8ecf11e675f35c5" Jul 15 14:39:26.000883 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.000792 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-28whz"] Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001194 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="pull" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001211 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="pull" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001229 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="util" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001237 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="util" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001252 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="extract" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001260 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="extract" Jul 15 14:39:26.001370 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.001324 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="847902fa-57b5-4332-a138-a46f7623f1da" containerName="extract" Jul 15 14:39:26.004669 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.004647 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.006780 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.006763 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 14:39:26.006945 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.006929 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8ccpk\"" Jul 15 14:39:26.007385 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.007369 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 14:39:26.018674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.018650 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-28whz"] Jul 15 14:39:26.122766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.122728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.122766 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.122764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k99\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-kube-api-access-n2k99\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.223739 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.223699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.223739 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.223738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k99\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-kube-api-access-n2k99\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.234163 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.234133 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.234610 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.234590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k99\" (UniqueName: \"kubernetes.io/projected/6dbf75dd-a2a9-4fac-8c1a-c2052ed54158-kube-api-access-n2k99\") pod \"cert-manager-webhook-6486ffd997-28whz\" (UID: \"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158\") " pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.314455 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.314424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:26.505915 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.505834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-28whz"] Jul 15 14:39:26.508511 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:39:26.508480 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dbf75dd_a2a9_4fac_8c1a_c2052ed54158.slice/crio-0167291632a7471f3c2a214ce5e372517062345d30d5ea499ebde2dbfd15abbe WatchSource:0}: Error finding container 0167291632a7471f3c2a214ce5e372517062345d30d5ea499ebde2dbfd15abbe: Status 404 returned error can't find the container with id 0167291632a7471f3c2a214ce5e372517062345d30d5ea499ebde2dbfd15abbe Jul 15 14:39:26.575230 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:26.575147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" event={"ID":"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158","Type":"ContainerStarted","Data":"0167291632a7471f3c2a214ce5e372517062345d30d5ea499ebde2dbfd15abbe"} Jul 15 14:39:30.597453 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:30.597415 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" event={"ID":"6dbf75dd-a2a9-4fac-8c1a-c2052ed54158","Type":"ContainerStarted","Data":"add55deef1a4f8b3567b6c1d96a2e5a27b90f726a8992a0aa222e44d2b964e06"} Jul 15 14:39:30.597815 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:30.597521 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:30.616671 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:30.616470 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" podStartSLOduration=1.6192021699999999 podStartE2EDuration="5.61644971s" totalImagesPullingTime="3.99724754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:30.615948857 +0000 UTC m=+341.025568136" watchObservedRunningTime="2026-07-15 14:39:30.61644971 +0000 UTC m=+341.026068989" Jul 15 14:39:31.299628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.299595 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6"] Jul 15 14:39:31.302686 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.302672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.305808 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.305781 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 14:39:31.306282 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.306267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-l6pmk\"" Jul 15 14:39:31.306618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.306599 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:39:31.313002 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.312981 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6"] Jul 15 14:39:31.464337 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.464296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft228\" (UniqueName: \"kubernetes.io/projected/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-kube-api-access-ft228\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.464520 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.464371 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.565421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.565337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft228\" (UniqueName: \"kubernetes.io/projected/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-kube-api-access-ft228\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.565421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.565388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.565700 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.565683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.576051 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.576024 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft228\" (UniqueName: \"kubernetes.io/projected/8b58961a-dd47-4cf5-8956-a8da8b3bbe26-kube-api-access-ft228\") pod \"openshift-lws-operator-fd8ccff4c-wlvd6\" (UID: \"8b58961a-dd47-4cf5-8956-a8da8b3bbe26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.611109 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.611084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" Jul 15 14:39:31.745443 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:31.745298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6"] Jul 15 14:39:31.747851 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:39:31.747788 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b58961a_dd47_4cf5_8956_a8da8b3bbe26.slice/crio-f1d7f99dcd658c6686d3d3f78571c495bc74b226b4e83b03ac484831bdd985b4 WatchSource:0}: Error finding container f1d7f99dcd658c6686d3d3f78571c495bc74b226b4e83b03ac484831bdd985b4: Status 404 returned error can't find the container with id f1d7f99dcd658c6686d3d3f78571c495bc74b226b4e83b03ac484831bdd985b4 Jul 15 14:39:32.606240 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:32.606206 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" event={"ID":"8b58961a-dd47-4cf5-8956-a8da8b3bbe26","Type":"ContainerStarted","Data":"f1d7f99dcd658c6686d3d3f78571c495bc74b226b4e83b03ac484831bdd985b4"} Jul 15 14:39:35.619327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:35.619244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" event={"ID":"8b58961a-dd47-4cf5-8956-a8da8b3bbe26","Type":"ContainerStarted","Data":"24d22724dfaa8fa424e6471af8f7bce6a503b596f1fd6169a8371e5661330c14"} Jul 15 14:39:35.641418 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:35.641367 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlvd6" podStartSLOduration=1.063775587 podStartE2EDuration="4.641351018s" totalImagesPullingTime="3.577575431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:35.638588627 +0000 UTC m=+346.048207900" watchObservedRunningTime="2026-07-15 14:39:35.641351018 +0000 UTC m=+346.050970287" Jul 15 14:39:36.603901 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:36.603867 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-28whz" Jul 15 14:39:38.236579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.236543 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7"] Jul 15 14:39:38.239178 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.239158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.241809 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.241792 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 14:39:38.241897 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.241792 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4cxzl\"" Jul 15 14:39:38.242275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.242263 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 14:39:38.253275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.253250 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7"] Jul 15 14:39:38.321571 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.321532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6mf\" (UniqueName: \"kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.321752 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.321580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.321752 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.321671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.422579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.422545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6mf\" (UniqueName: \"kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.422760 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.422590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.422760 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.422644 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.423106 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.423085 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.423106 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.423100 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.432299 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.432275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6mf\" (UniqueName: \"kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.548451 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.548355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:38.696442 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:38.696417 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7"] Jul 15 14:39:38.698689 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:39:38.698662 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88923e7b_3476_47ce_a74f_32bf0a8603c5.slice/crio-6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14 WatchSource:0}: Error finding container 6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14: Status 404 returned error can't find the container with id 6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14 Jul 15 14:39:39.635671 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:39.635637 2567 generic.go:356] "Generic (PLEG): container finished" podID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerID="0ebe21c2f13451bd89e7a3c63d635bf4d25d67be7803a9769db120080a80cb2e" exitCode=0 Jul 15 14:39:39.636100 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:39.635725 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" event={"ID":"88923e7b-3476-47ce-a74f-32bf0a8603c5","Type":"ContainerDied","Data":"0ebe21c2f13451bd89e7a3c63d635bf4d25d67be7803a9769db120080a80cb2e"} Jul 15 14:39:39.636100 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:39.635776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" event={"ID":"88923e7b-3476-47ce-a74f-32bf0a8603c5","Type":"ContainerStarted","Data":"6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14"} Jul 15 14:39:41.645045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:41.645006 2567 generic.go:356] "Generic (PLEG): container finished" podID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerID="a4f28735dcaf427c92cdf9033114368c6dccc2c4f731280a4725914ceb225421" exitCode=0 Jul 15 14:39:41.645457 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:41.645088 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" event={"ID":"88923e7b-3476-47ce-a74f-32bf0a8603c5","Type":"ContainerDied","Data":"a4f28735dcaf427c92cdf9033114368c6dccc2c4f731280a4725914ceb225421"} Jul 15 14:39:42.650755 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:42.650722 2567 generic.go:356] "Generic (PLEG): container finished" podID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerID="97a6dcde3442956b001c77f1181616fe0f3a5167f9021a34daaae70faedcfc01" exitCode=0 Jul 15 14:39:42.651140 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:42.650792 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" event={"ID":"88923e7b-3476-47ce-a74f-32bf0a8603c5","Type":"ContainerDied","Data":"97a6dcde3442956b001c77f1181616fe0f3a5167f9021a34daaae70faedcfc01"} Jul 15 14:39:43.774639 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.774614 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:43.967228 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.967136 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util\") pod \"88923e7b-3476-47ce-a74f-32bf0a8603c5\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " Jul 15 14:39:43.967228 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.967195 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq6mf\" (UniqueName: \"kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf\") pod \"88923e7b-3476-47ce-a74f-32bf0a8603c5\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " Jul 15 14:39:43.967228 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.967217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle\") pod \"88923e7b-3476-47ce-a74f-32bf0a8603c5\" (UID: \"88923e7b-3476-47ce-a74f-32bf0a8603c5\") " Jul 15 14:39:43.968010 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.967979 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle" (OuterVolumeSpecName: "bundle") pod "88923e7b-3476-47ce-a74f-32bf0a8603c5" (UID: "88923e7b-3476-47ce-a74f-32bf0a8603c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:43.969301 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.969274 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf" (OuterVolumeSpecName: "kube-api-access-lq6mf") pod "88923e7b-3476-47ce-a74f-32bf0a8603c5" (UID: "88923e7b-3476-47ce-a74f-32bf0a8603c5"). InnerVolumeSpecName "kube-api-access-lq6mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:39:43.972344 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:43.972308 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util" (OuterVolumeSpecName: "util") pod "88923e7b-3476-47ce-a74f-32bf0a8603c5" (UID: "88923e7b-3476-47ce-a74f-32bf0a8603c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:39:44.068402 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.068355 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6mf\" (UniqueName: \"kubernetes.io/projected/88923e7b-3476-47ce-a74f-32bf0a8603c5-kube-api-access-lq6mf\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:44.068402 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.068390 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:44.068402 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.068399 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88923e7b-3476-47ce-a74f-32bf0a8603c5-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:39:44.660644 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.660611 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" event={"ID":"88923e7b-3476-47ce-a74f-32bf0a8603c5","Type":"ContainerDied","Data":"6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14"} Jul 15 14:39:44.660644 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.660651 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbqzg7" Jul 15 14:39:44.660845 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:44.660662 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db197d5f79945bead682cbcc9c1154fdad824699f542bfe968517aefc6a9d14" Jul 15 14:39:54.279048 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279019 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt"] Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279288 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="util" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279299 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="util" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279308 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="pull" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279313 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="pull" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279322 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="extract" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279328 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="extract" Jul 15 14:39:54.279405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.279370 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="88923e7b-3476-47ce-a74f-32bf0a8603c5" containerName="extract" Jul 15 14:39:54.282236 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.282221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.286041 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.286021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 14:39:54.293621 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.293603 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 14:39:54.293718 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.293646 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 14:39:54.294428 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.294414 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vdrdb\"" Jul 15 14:39:54.297776 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.297764 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 14:39:54.324321 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.324299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt"] Jul 15 14:39:54.349720 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.349698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.349811 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.349726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.349811 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.349743 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.349811 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.349757 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzc5\" (UniqueName: \"kubernetes.io/projected/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-kube-api-access-skzc5\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.349811 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.349782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.450856 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.450831 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.450955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.450867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.450955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.450895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.450955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.450917 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzc5\" (UniqueName: \"kubernetes.io/projected/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-kube-api-access-skzc5\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.450955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.450946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.451244 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.451223 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.453176 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.453152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.453272 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.453229 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.453371 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.453354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.474933 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.474909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzc5\" (UniqueName: \"kubernetes.io/projected/65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d-kube-api-access-skzc5\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt\" (UID: \"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.591428 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.591409 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:54.728763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:54.728734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt"] Jul 15 14:39:54.730041 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:39:54.730016 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65424aea_3b5b_490c_ac9f_f2a1a6cb5e4d.slice/crio-26bd13601d1eca601802942b47f0cd68e5578805d4585f5b1ddc0e7ca3e3f8d5 WatchSource:0}: Error finding container 26bd13601d1eca601802942b47f0cd68e5578805d4585f5b1ddc0e7ca3e3f8d5: Status 404 returned error can't find the container with id 26bd13601d1eca601802942b47f0cd68e5578805d4585f5b1ddc0e7ca3e3f8d5 Jul 15 14:39:55.709145 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:55.709109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" event={"ID":"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d","Type":"ContainerStarted","Data":"26bd13601d1eca601802942b47f0cd68e5578805d4585f5b1ddc0e7ca3e3f8d5"} Jul 15 14:39:57.719351 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:57.719322 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" event={"ID":"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d","Type":"ContainerStarted","Data":"5c7d6a38a17b5c920263c524f0bf0464323be7c3f190b8d2d47aa6fc84ab1cb8"} Jul 15 14:39:58.724358 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:58.724331 2567 generic.go:356] "Generic (PLEG): container finished" podID="65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d" containerID="5c7d6a38a17b5c920263c524f0bf0464323be7c3f190b8d2d47aa6fc84ab1cb8" exitCode=0 Jul 15 14:39:58.724696 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:58.724405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" event={"ID":"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d","Type":"ContainerDied","Data":"5c7d6a38a17b5c920263c524f0bf0464323be7c3f190b8d2d47aa6fc84ab1cb8"} Jul 15 14:39:59.731658 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:59.731621 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" event={"ID":"65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d","Type":"ContainerStarted","Data":"6f878692c9b2f0512320529367a9fa9eb54ac841dc64e12d0759ff1c159225aa"} Jul 15 14:39:59.732123 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:59.731762 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:39:59.768256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:39:59.768077 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" podStartSLOduration=1.159351379 podStartE2EDuration="5.768035868s" totalImagesPullingTime="3.544850238s" totalInitContainerRuntime="1.063834251s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:59.766709683 +0000 UTC m=+370.176328961" watchObservedRunningTime="2026-07-15 14:39:59.768035868 +0000 UTC m=+370.177655146" Jul 15 14:40:10.738642 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:10.738605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt" Jul 15 14:40:16.221771 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.221738 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b75mx"] Jul 15 14:40:16.226275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.226253 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.229744 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.229726 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 14:40:16.230500 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.230481 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-64bts\"" Jul 15 14:40:16.237356 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.237331 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b75mx"] Jul 15 14:40:16.310922 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.310888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6t6\" (UniqueName: \"kubernetes.io/projected/686bb40f-25d6-4b47-a59f-e307d4e306bb-kube-api-access-hw6t6\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.311108 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.310942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.412001 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.411966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.412194 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.412067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6t6\" (UniqueName: \"kubernetes.io/projected/686bb40f-25d6-4b47-a59f-e307d4e306bb-kube-api-access-hw6t6\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.412194 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:40:16.412147 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 14:40:16.412273 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:40:16.412225 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert podName:686bb40f-25d6-4b47-a59f-e307d4e306bb nodeName:}" failed. No retries permitted until 2026-07-15 14:40:16.912206345 +0000 UTC m=+387.321825602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert") pod "odh-model-controller-7998c67dc4-b75mx" (UID: "686bb40f-25d6-4b47-a59f-e307d4e306bb") : secret "odh-model-controller-webhook-cert" not found Jul 15 14:40:16.422469 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.422441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6t6\" (UniqueName: \"kubernetes.io/projected/686bb40f-25d6-4b47-a59f-e307d4e306bb-kube-api-access-hw6t6\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.916200 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.916160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:16.918503 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:16.918478 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686bb40f-25d6-4b47-a59f-e307d4e306bb-cert\") pod \"odh-model-controller-7998c67dc4-b75mx\" (UID: \"686bb40f-25d6-4b47-a59f-e307d4e306bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:17.036733 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.036703 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8td92"] Jul 15 14:40:17.040024 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.040007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.042177 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.042155 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 14:40:17.042289 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.042233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-q7mrw\"" Jul 15 14:40:17.074751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.074726 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8td92"] Jul 15 14:40:17.117845 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.117820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.117957 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.117853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852sd\" (UniqueName: \"kubernetes.io/projected/605132e6-3ba0-40a3-97a6-db746c1c171e-kube-api-access-852sd\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.136852 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.136824 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:17.218960 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.218934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.219162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.218970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852sd\" (UniqueName: \"kubernetes.io/projected/605132e6-3ba0-40a3-97a6-db746c1c171e-kube-api-access-852sd\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.219814 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:40:17.219321 2567 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 15 14:40:17.219814 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:40:17.219371 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs podName:605132e6-3ba0-40a3-97a6-db746c1c171e nodeName:}" failed. No retries permitted until 2026-07-15 14:40:17.71935629 +0000 UTC m=+388.128975546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs") pod "model-serving-api-54d89bd5fc-8td92" (UID: "605132e6-3ba0-40a3-97a6-db746c1c171e") : secret "model-serving-api-tls" not found Jul 15 14:40:17.250903 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.250849 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852sd\" (UniqueName: \"kubernetes.io/projected/605132e6-3ba0-40a3-97a6-db746c1c171e-kube-api-access-852sd\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.487105 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.487006 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b75mx"] Jul 15 14:40:17.499090 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:17.499040 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686bb40f_25d6_4b47_a59f_e307d4e306bb.slice/crio-d032dafa4d0321c8c184329927d1a02805490090c03ce5f634c9fcc349de0f92 WatchSource:0}: Error finding container d032dafa4d0321c8c184329927d1a02805490090c03ce5f634c9fcc349de0f92: Status 404 returned error can't find the container with id d032dafa4d0321c8c184329927d1a02805490090c03ce5f634c9fcc349de0f92 Jul 15 14:40:17.723576 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.723544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.725810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.725791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605132e6-3ba0-40a3-97a6-db746c1c171e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8td92\" (UID: \"605132e6-3ba0-40a3-97a6-db746c1c171e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:17.814580 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.814542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" event={"ID":"686bb40f-25d6-4b47-a59f-e307d4e306bb","Type":"ContainerStarted","Data":"d032dafa4d0321c8c184329927d1a02805490090c03ce5f634c9fcc349de0f92"} Jul 15 14:40:17.953649 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:17.953613 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:18.101664 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:18.101641 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8td92"] Jul 15 14:40:18.103923 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:18.103895 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605132e6_3ba0_40a3_97a6_db746c1c171e.slice/crio-fe42a540304d179733e810d466c23131aef498257c9bb288a2dd52f6da9aa0ef WatchSource:0}: Error finding container fe42a540304d179733e810d466c23131aef498257c9bb288a2dd52f6da9aa0ef: Status 404 returned error can't find the container with id fe42a540304d179733e810d466c23131aef498257c9bb288a2dd52f6da9aa0ef Jul 15 14:40:18.820199 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:18.820166 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" event={"ID":"605132e6-3ba0-40a3-97a6-db746c1c171e","Type":"ContainerStarted","Data":"fe42a540304d179733e810d466c23131aef498257c9bb288a2dd52f6da9aa0ef"} Jul 15 14:40:22.843110 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.843069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" event={"ID":"686bb40f-25d6-4b47-a59f-e307d4e306bb","Type":"ContainerStarted","Data":"8fd45c80f510d07b6720d6f58e8296f65e85afc44332449a3b1054098f870da8"} Jul 15 14:40:22.843595 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.843234 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:22.844892 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.844866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" event={"ID":"605132e6-3ba0-40a3-97a6-db746c1c171e","Type":"ContainerStarted","Data":"aaa77add1ab8d8a47a2fcfb6da46e76b6eb3d070f3c99c96267a774241b5aa9f"} Jul 15 14:40:22.845015 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.845003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:22.880513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.880461 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" podStartSLOduration=2.54423451 podStartE2EDuration="6.880443565s" totalImagesPullingTime="4.336209055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:22.875847838 +0000 UTC m=+393.285467131" watchObservedRunningTime="2026-07-15 14:40:22.880443565 +0000 UTC m=+393.290062843" Jul 15 14:40:22.904317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:22.904266 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" podStartSLOduration=2.220925237 podStartE2EDuration="5.904251266s" totalImagesPullingTime="3.683326029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:22.903018409 +0000 UTC m=+393.312637686" watchObservedRunningTime="2026-07-15 14:40:22.904251266 +0000 UTC m=+393.313870545" Jul 15 14:40:27.160817 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.160786 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85d556468c-t8vkk"] Jul 15 14:40:27.163627 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.163603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.170262 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.170237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 14:40:27.170372 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.170288 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ld2h7\"" Jul 15 14:40:27.170686 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.170671 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 14:40:27.170748 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.170703 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 14:40:27.170938 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.170926 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 14:40:27.182941 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.182919 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d556468c-t8vkk"] Jul 15 14:40:27.299368 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.299341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c26196e2-66c6-474a-9eb9-17268e4dba2f-tls-certs\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.299450 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.299406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26196e2-66c6-474a-9eb9-17268e4dba2f-tmp\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.299600 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.299583 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57czw\" (UniqueName: \"kubernetes.io/projected/c26196e2-66c6-474a-9eb9-17268e4dba2f-kube-api-access-57czw\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.400583 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.400563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c26196e2-66c6-474a-9eb9-17268e4dba2f-tls-certs\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.400670 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.400604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26196e2-66c6-474a-9eb9-17268e4dba2f-tmp\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.400670 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.400625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57czw\" (UniqueName: \"kubernetes.io/projected/c26196e2-66c6-474a-9eb9-17268e4dba2f-kube-api-access-57czw\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.402644 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.402622 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26196e2-66c6-474a-9eb9-17268e4dba2f-tmp\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.402847 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.402830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c26196e2-66c6-474a-9eb9-17268e4dba2f-tls-certs\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.411824 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.411778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57czw\" (UniqueName: \"kubernetes.io/projected/c26196e2-66c6-474a-9eb9-17268e4dba2f-kube-api-access-57czw\") pod \"kube-auth-proxy-85d556468c-t8vkk\" (UID: \"c26196e2-66c6-474a-9eb9-17268e4dba2f\") " pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.472955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.472933 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" Jul 15 14:40:27.610884 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.610856 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d556468c-t8vkk"] Jul 15 14:40:27.614361 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:27.614336 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26196e2_66c6_474a_9eb9_17268e4dba2f.slice/crio-1bdd03bf1ab6953664f2a1809b4a3cc612a2ef39d32d8a36826495ce7520aab3 WatchSource:0}: Error finding container 1bdd03bf1ab6953664f2a1809b4a3cc612a2ef39d32d8a36826495ce7520aab3: Status 404 returned error can't find the container with id 1bdd03bf1ab6953664f2a1809b4a3cc612a2ef39d32d8a36826495ce7520aab3 Jul 15 14:40:27.785007 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.784929 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6nccc"] Jul 15 14:40:27.798191 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.798167 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:27.802329 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.802287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 14:40:27.802496 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.802391 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-snjcw\"" Jul 15 14:40:27.810482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.810454 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h"] Jul 15 14:40:27.821971 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.821951 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6nccc"] Jul 15 14:40:27.822082 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.822036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:27.825216 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.824791 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wpk8s\"" Jul 15 14:40:27.825216 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.825042 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 14:40:27.836338 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.836197 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h"] Jul 15 14:40:27.867110 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.867083 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" event={"ID":"c26196e2-66c6-474a-9eb9-17268e4dba2f","Type":"ContainerStarted","Data":"1bdd03bf1ab6953664f2a1809b4a3cc612a2ef39d32d8a36826495ce7520aab3"} Jul 15 14:40:27.906614 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.906580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7fr\" (UniqueName: \"kubernetes.io/projected/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-kube-api-access-nm7fr\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:27.906763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.906628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:27.906763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.906650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab6a9a40-9a37-451b-a989-245300788082-cert\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:27.906763 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:27.906712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tp7c\" (UniqueName: \"kubernetes.io/projected/ab6a9a40-9a37-451b-a989-245300788082-kube-api-access-5tp7c\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.008116 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.008053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7fr\" (UniqueName: \"kubernetes.io/projected/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-kube-api-access-nm7fr\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:28.008116 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.008123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:28.008343 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.008142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab6a9a40-9a37-451b-a989-245300788082-cert\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.008343 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.008178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tp7c\" (UniqueName: \"kubernetes.io/projected/ab6a9a40-9a37-451b-a989-245300788082-kube-api-access-5tp7c\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.010559 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.010527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:28.010673 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.010566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab6a9a40-9a37-451b-a989-245300788082-cert\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.021345 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.021322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7fr\" (UniqueName: \"kubernetes.io/projected/7c8e7cd2-c069-4fc8-be78-bfbf10517c69-kube-api-access-nm7fr\") pod \"llmisvc-controller-manager-777c6bd7d5-t9r6h\" (UID: \"7c8e7cd2-c069-4fc8-be78-bfbf10517c69\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:28.021563 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.021539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tp7c\" (UniqueName: \"kubernetes.io/projected/ab6a9a40-9a37-451b-a989-245300788082-kube-api-access-5tp7c\") pod \"kserve-controller-manager-d6f4b65b8-6nccc\" (UID: \"ab6a9a40-9a37-451b-a989-245300788082\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.113705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.113672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:28.138628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.138598 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:28.298945 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.298924 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6nccc"] Jul 15 14:40:28.300999 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:28.300974 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6a9a40_9a37_451b_a989_245300788082.slice/crio-315cb6c24a302a617f75a647e521d9f1d635103b2e8006269f9356e467bc489c WatchSource:0}: Error finding container 315cb6c24a302a617f75a647e521d9f1d635103b2e8006269f9356e467bc489c: Status 404 returned error can't find the container with id 315cb6c24a302a617f75a647e521d9f1d635103b2e8006269f9356e467bc489c Jul 15 14:40:28.316546 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:28.316520 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c8e7cd2_c069_4fc8_be78_bfbf10517c69.slice/crio-6e9c94df9d74d9b1a390be5b61f86b8eb97f2a678bebe585194b94b00f5bb1df WatchSource:0}: Error finding container 6e9c94df9d74d9b1a390be5b61f86b8eb97f2a678bebe585194b94b00f5bb1df: Status 404 returned error can't find the container with id 6e9c94df9d74d9b1a390be5b61f86b8eb97f2a678bebe585194b94b00f5bb1df Jul 15 14:40:28.320936 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.320917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h"] Jul 15 14:40:28.874818 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.874782 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" event={"ID":"7c8e7cd2-c069-4fc8-be78-bfbf10517c69","Type":"ContainerStarted","Data":"6e9c94df9d74d9b1a390be5b61f86b8eb97f2a678bebe585194b94b00f5bb1df"} Jul 15 14:40:28.876078 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:28.876031 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" event={"ID":"ab6a9a40-9a37-451b-a989-245300788082","Type":"ContainerStarted","Data":"315cb6c24a302a617f75a647e521d9f1d635103b2e8006269f9356e467bc489c"} Jul 15 14:40:29.758648 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.758610 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv"] Jul 15 14:40:29.762179 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.762137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.768241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.766598 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 14:40:29.768241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.766822 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 14:40:29.768241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.767416 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-t2s9q\"" Jul 15 14:40:29.768241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.767609 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 14:40:29.771247 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.771206 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 14:40:29.801570 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.801495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv"] Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2xf\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-kube-api-access-4b2xf\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827624 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.827743 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.827706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929781 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929747 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2xf\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-kube-api-access-4b2xf\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.929984 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.929989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.930327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.930032 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.930327 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.930266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.932962 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.932935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.933894 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.933873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.934373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.934345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.936038 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.935983 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.950140 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.950112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:29.950702 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:29.950666 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2xf\" (UniqueName: \"kubernetes.io/projected/445e192e-a9b9-4198-82c9-0e5ea71d9dc7-kube-api-access-4b2xf\") pod \"istiod-openshift-gateway-94bb8fbfd-dg7kv\" (UID: \"445e192e-a9b9-4198-82c9-0e5ea71d9dc7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:30.081012 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:30.080416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:30.300792 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:30.300427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv"] Jul 15 14:40:30.856709 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:40:30.856679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445e192e_a9b9_4198_82c9_0e5ea71d9dc7.slice/crio-23761c312096c286ec40600465efa33341c9a64c4809dd838337e862f5f03ff3 WatchSource:0}: Error finding container 23761c312096c286ec40600465efa33341c9a64c4809dd838337e862f5f03ff3: Status 404 returned error can't find the container with id 23761c312096c286ec40600465efa33341c9a64c4809dd838337e862f5f03ff3 Jul 15 14:40:30.888326 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:30.888298 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" event={"ID":"445e192e-a9b9-4198-82c9-0e5ea71d9dc7","Type":"ContainerStarted","Data":"23761c312096c286ec40600465efa33341c9a64c4809dd838337e862f5f03ff3"} Jul 15 14:40:32.906030 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.905991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" event={"ID":"c26196e2-66c6-474a-9eb9-17268e4dba2f","Type":"ContainerStarted","Data":"c9db5d2c0a56d89ef4c4307d4bdba0bcf730c47f35b44f77b3eee5afa4eff59c"} Jul 15 14:40:32.908091 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.908040 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" event={"ID":"7c8e7cd2-c069-4fc8-be78-bfbf10517c69","Type":"ContainerStarted","Data":"37e4831d39ec87d5a3d99f6378216ce6e3ef483eb2e2652614cd20c7d8341a29"} Jul 15 14:40:32.908232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.908142 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:40:32.910190 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.910165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" event={"ID":"ab6a9a40-9a37-451b-a989-245300788082","Type":"ContainerStarted","Data":"70c6022ef913c763b9c518f703580716e0121fc1c50813709a0ab487943f27a5"} Jul 15 14:40:32.910298 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.910288 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:40:32.930581 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.930527 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85d556468c-t8vkk" podStartSLOduration=1.415182761 podStartE2EDuration="5.930508965s" totalImagesPullingTime="4.515326204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:32.928706013 +0000 UTC m=+403.338325318" watchObservedRunningTime="2026-07-15 14:40:32.930508965 +0000 UTC m=+403.340128243" Jul 15 14:40:32.986326 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.986265 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" podStartSLOduration=2.162329352 podStartE2EDuration="5.986241618s" totalImagesPullingTime="3.823912266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:32.985299095 +0000 UTC m=+403.394918374" watchObservedRunningTime="2026-07-15 14:40:32.986241618 +0000 UTC m=+403.395860899" Jul 15 14:40:32.987042 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:32.987010 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" podStartSLOduration=2.173068842 podStartE2EDuration="5.986998505s" totalImagesPullingTime="3.813929663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:32.95419071 +0000 UTC m=+403.363809989" watchObservedRunningTime="2026-07-15 14:40:32.986998505 +0000 UTC m=+403.396617786" Jul 15 14:40:33.851577 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:33.851544 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-b75mx" Jul 15 14:40:33.854866 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:33.854563 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8td92" Jul 15 14:40:33.967588 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:33.967557 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 14:40:33.967875 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:33.967616 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 14:40:34.924127 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:34.924086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" event={"ID":"445e192e-a9b9-4198-82c9-0e5ea71d9dc7","Type":"ContainerStarted","Data":"96178ecf5db71391bbfedb0b405bab4c37e696f2bce550257aba2046d846e82e"} Jul 15 14:40:34.924532 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:34.924504 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:40:34.951225 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:34.951162 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" podStartSLOduration=2.8427354989999998 podStartE2EDuration="5.951142353s" totalImagesPullingTime="3.108406854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:34.950147906 +0000 UTC m=+405.359767183" watchObservedRunningTime="2026-07-15 14:40:34.951142353 +0000 UTC m=+405.360761633" Jul 15 14:40:35.931002 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:35.930958 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-dg7kv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 14:40:35.931436 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:35.931020 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" podUID="445e192e-a9b9-4198-82c9-0e5ea71d9dc7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 14:40:38.930256 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:40:38.930228 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dg7kv" Jul 15 14:41:03.917618 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:03.917589 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t9r6h" Jul 15 14:41:03.920584 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:03.920568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6nccc" Jul 15 14:41:32.989026 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:32.988983 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt"] Jul 15 14:41:32.991843 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:32.991824 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:32.994477 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:32.994451 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hxgsr\"" Jul 15 14:41:32.994641 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:32.994614 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 14:41:32.994987 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:32.994970 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 14:41:33.006909 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.006884 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt"] Jul 15 14:41:33.149782 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.149753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.149882 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.149788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.149882 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.149844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbw7\" (UniqueName: \"kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.250858 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.250797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.250858 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.250831 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.250858 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.250855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbw7\" (UniqueName: \"kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.251173 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.251154 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.251222 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.251195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.261411 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.261387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbw7\" (UniqueName: \"kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.301310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.301289 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:33.437915 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.437892 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt"] Jul 15 14:41:33.440278 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:33.440251 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61340d45_9b18_4baf_8ded_667e3bf32e08.slice/crio-f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78 WatchSource:0}: Error finding container f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78: Status 404 returned error can't find the container with id f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78 Jul 15 14:41:33.589787 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.589760 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn"] Jul 15 14:41:33.592195 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.592180 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.604834 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.604811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn"] Jul 15 14:41:33.755308 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.755284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqbl\" (UniqueName: \"kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.755411 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.755322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.755411 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.755365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.856386 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.856357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.856513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.856406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqbl\" (UniqueName: \"kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.856513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.856441 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.856774 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.856754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.856811 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.856794 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.869792 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.869769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqbl\" (UniqueName: \"kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:33.910282 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:33.910262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:34.052604 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.052573 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl"] Jul 15 14:41:34.055932 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.055914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn"] Jul 15 14:41:34.056046 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.056035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.056802 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:34.056780 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac858ac_67c1_4bc2_bae5_cf5515a5ac73.slice/crio-0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e WatchSource:0}: Error finding container 0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e: Status 404 returned error can't find the container with id 0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e Jul 15 14:41:34.062201 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.062119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl"] Jul 15 14:41:34.159148 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.159121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.159281 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.159164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7xm\" (UniqueName: \"kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.159340 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.159284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.200158 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.200123 2567 generic.go:356] "Generic (PLEG): container finished" podID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerID="94626cc7e1e65128b6e022517b8a62886a9d131a3914e7a2d424c3854f8ac473" exitCode=0 Jul 15 14:41:34.200275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.200201 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" event={"ID":"aac858ac-67c1-4bc2-bae5-cf5515a5ac73","Type":"ContainerDied","Data":"94626cc7e1e65128b6e022517b8a62886a9d131a3914e7a2d424c3854f8ac473"} Jul 15 14:41:34.200275 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.200249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" event={"ID":"aac858ac-67c1-4bc2-bae5-cf5515a5ac73","Type":"ContainerStarted","Data":"0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e"} Jul 15 14:41:34.201804 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.201787 2567 generic.go:356] "Generic (PLEG): container finished" podID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerID="345500ef739b6e111e0c394d131489f21b49d07dde30135d7d598796b4a8d3db" exitCode=0 Jul 15 14:41:34.201895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.201835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" event={"ID":"61340d45-9b18-4baf-8ded-667e3bf32e08","Type":"ContainerDied","Data":"345500ef739b6e111e0c394d131489f21b49d07dde30135d7d598796b4a8d3db"} Jul 15 14:41:34.201895 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.201862 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" event={"ID":"61340d45-9b18-4baf-8ded-667e3bf32e08","Type":"ContainerStarted","Data":"f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78"} Jul 15 14:41:34.260719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.260694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.260840 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.260738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.260887 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.260857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7xm\" (UniqueName: \"kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.261031 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.261017 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.261123 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.261103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.271364 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.271339 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7xm\" (UniqueName: \"kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.366554 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.366500 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:34.382753 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.382729 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5"] Jul 15 14:41:34.385393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.385378 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.397333 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.397311 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5"] Jul 15 14:41:34.462571 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.462542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.462705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.462600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.462705 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.462666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wl9\" (UniqueName: \"kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.505171 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.505143 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl"] Jul 15 14:41:34.505967 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:34.505945 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe2e190_a4f2_4158_89d6_0c85c11f3d41.slice/crio-e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d WatchSource:0}: Error finding container e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d: Status 404 returned error can't find the container with id e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d Jul 15 14:41:34.563218 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.563197 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wl9\" (UniqueName: \"kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.563299 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.563260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.563299 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.563286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.563551 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.563536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.563619 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.563599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.573405 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.573386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wl9\" (UniqueName: \"kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.695179 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.695129 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:34.827173 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:34.827151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5"] Jul 15 14:41:34.832982 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:34.832960 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834e72c5_f717_464f_ab98_94b84775339e.slice/crio-c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e WatchSource:0}: Error finding container c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e: Status 404 returned error can't find the container with id c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e Jul 15 14:41:35.211181 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.211147 2567 generic.go:356] "Generic (PLEG): container finished" podID="834e72c5-f717-464f-ab98-94b84775339e" containerID="8482f68231fab2df9064b132a4dc9ba6929da357e9a2f8b2d44359fe0151c5fc" exitCode=0 Jul 15 14:41:35.211541 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.211227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" event={"ID":"834e72c5-f717-464f-ab98-94b84775339e","Type":"ContainerDied","Data":"8482f68231fab2df9064b132a4dc9ba6929da357e9a2f8b2d44359fe0151c5fc"} Jul 15 14:41:35.211541 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.211279 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" event={"ID":"834e72c5-f717-464f-ab98-94b84775339e","Type":"ContainerStarted","Data":"c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e"} Jul 15 14:41:35.219628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.219603 2567 generic.go:356] "Generic (PLEG): container finished" podID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerID="12e25770ac428fe71837b5c27f80f0da547a805420415eee1feaf963731f2abf" exitCode=0 Jul 15 14:41:35.219747 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.219711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" event={"ID":"61340d45-9b18-4baf-8ded-667e3bf32e08","Type":"ContainerDied","Data":"12e25770ac428fe71837b5c27f80f0da547a805420415eee1feaf963731f2abf"} Jul 15 14:41:35.221231 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.221198 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerID="69cd1db23a2c50ddc3c92377ca499ce2b4f23b9c7103b1a86adb0be520941efa" exitCode=0 Jul 15 14:41:35.221319 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.221254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" event={"ID":"7fe2e190-a4f2-4158-89d6-0c85c11f3d41","Type":"ContainerDied","Data":"69cd1db23a2c50ddc3c92377ca499ce2b4f23b9c7103b1a86adb0be520941efa"} Jul 15 14:41:35.221319 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:35.221300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" event={"ID":"7fe2e190-a4f2-4158-89d6-0c85c11f3d41","Type":"ContainerStarted","Data":"e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d"} Jul 15 14:41:36.231226 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:36.231197 2567 generic.go:356] "Generic (PLEG): container finished" podID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerID="82dac058a74dddacc013eda41a6dce09071ef702b7d9d257874ec36736e92592" exitCode=0 Jul 15 14:41:36.231601 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:36.231264 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" event={"ID":"61340d45-9b18-4baf-8ded-667e3bf32e08","Type":"ContainerDied","Data":"82dac058a74dddacc013eda41a6dce09071ef702b7d9d257874ec36736e92592"} Jul 15 14:41:37.238780 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.238755 2567 generic.go:356] "Generic (PLEG): container finished" podID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerID="5f1a8ae767bb15f8e6d5a0e1ba066a7a7312da6b2a552efee80373221d948510" exitCode=0 Jul 15 14:41:37.239140 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.238829 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" event={"ID":"aac858ac-67c1-4bc2-bae5-cf5515a5ac73","Type":"ContainerDied","Data":"5f1a8ae767bb15f8e6d5a0e1ba066a7a7312da6b2a552efee80373221d948510"} Jul 15 14:41:37.241132 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.241109 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerID="d6e03491135db3646aa286c2fdc8e00023c76622a1ad04cb278ad49e23123c47" exitCode=0 Jul 15 14:41:37.241253 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.241199 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" event={"ID":"7fe2e190-a4f2-4158-89d6-0c85c11f3d41","Type":"ContainerDied","Data":"d6e03491135db3646aa286c2fdc8e00023c76622a1ad04cb278ad49e23123c47"} Jul 15 14:41:37.494699 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.494655 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:37.587344 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.587321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util\") pod \"61340d45-9b18-4baf-8ded-667e3bf32e08\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " Jul 15 14:41:37.587436 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.587387 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzbw7\" (UniqueName: \"kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7\") pod \"61340d45-9b18-4baf-8ded-667e3bf32e08\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " Jul 15 14:41:37.587436 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.587414 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle\") pod \"61340d45-9b18-4baf-8ded-667e3bf32e08\" (UID: \"61340d45-9b18-4baf-8ded-667e3bf32e08\") " Jul 15 14:41:37.587896 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.587867 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle" (OuterVolumeSpecName: "bundle") pod "61340d45-9b18-4baf-8ded-667e3bf32e08" (UID: "61340d45-9b18-4baf-8ded-667e3bf32e08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:37.589485 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.589468 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7" (OuterVolumeSpecName: "kube-api-access-zzbw7") pod "61340d45-9b18-4baf-8ded-667e3bf32e08" (UID: "61340d45-9b18-4baf-8ded-667e3bf32e08"). InnerVolumeSpecName "kube-api-access-zzbw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:41:37.592918 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.592897 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util" (OuterVolumeSpecName: "util") pod "61340d45-9b18-4baf-8ded-667e3bf32e08" (UID: "61340d45-9b18-4baf-8ded-667e3bf32e08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:37.688876 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.688856 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:37.688876 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.688877 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbw7\" (UniqueName: \"kubernetes.io/projected/61340d45-9b18-4baf-8ded-667e3bf32e08-kube-api-access-zzbw7\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:37.688990 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:37.688887 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61340d45-9b18-4baf-8ded-667e3bf32e08-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:38.248675 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.248652 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" Jul 15 14:41:38.248675 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.248657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt" event={"ID":"61340d45-9b18-4baf-8ded-667e3bf32e08","Type":"ContainerDied","Data":"f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78"} Jul 15 14:41:38.249043 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.248707 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e1c844615d1201566341c388fa3741ab9ad261f74aba4172b6c97c6d6eda78" Jul 15 14:41:38.251202 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.251177 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerID="a856b03935e6f8fa6c29652d2ab1b5195ad3647722028578396c4cd4b95c29b1" exitCode=0 Jul 15 14:41:38.251300 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.251243 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" event={"ID":"7fe2e190-a4f2-4158-89d6-0c85c11f3d41","Type":"ContainerDied","Data":"a856b03935e6f8fa6c29652d2ab1b5195ad3647722028578396c4cd4b95c29b1"} Jul 15 14:41:38.253398 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.253378 2567 generic.go:356] "Generic (PLEG): container finished" podID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerID="aa4583a7d6a524e4390873946ba21ca720459339997f6a2640db8c1f9c2a2250" exitCode=0 Jul 15 14:41:38.253482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.253448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" event={"ID":"aac858ac-67c1-4bc2-bae5-cf5515a5ac73","Type":"ContainerDied","Data":"aa4583a7d6a524e4390873946ba21ca720459339997f6a2640db8c1f9c2a2250"} Jul 15 14:41:38.255182 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.255162 2567 generic.go:356] "Generic (PLEG): container finished" podID="834e72c5-f717-464f-ab98-94b84775339e" containerID="f565a24c1327361c7b33f514e9f9bcb6ba3b1a1fa16a865b237413c984949164" exitCode=0 Jul 15 14:41:38.255269 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:38.255192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" event={"ID":"834e72c5-f717-464f-ab98-94b84775339e","Type":"ContainerDied","Data":"f565a24c1327361c7b33f514e9f9bcb6ba3b1a1fa16a865b237413c984949164"} Jul 15 14:41:39.262366 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.262330 2567 generic.go:356] "Generic (PLEG): container finished" podID="834e72c5-f717-464f-ab98-94b84775339e" containerID="4be1e17e837089ab2947f40b04b62a1954680b4031332b52c8dedcfbe284a9e7" exitCode=0 Jul 15 14:41:39.262782 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.262423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" event={"ID":"834e72c5-f717-464f-ab98-94b84775339e","Type":"ContainerDied","Data":"4be1e17e837089ab2947f40b04b62a1954680b4031332b52c8dedcfbe284a9e7"} Jul 15 14:41:39.403092 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.403039 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:39.426115 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.426096 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:39.503894 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.503871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util\") pod \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " Jul 15 14:41:39.504045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.503923 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmqbl\" (UniqueName: \"kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl\") pod \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " Jul 15 14:41:39.504045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.503945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util\") pod \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " Jul 15 14:41:39.504045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.503990 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle\") pod \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\" (UID: \"aac858ac-67c1-4bc2-bae5-cf5515a5ac73\") " Jul 15 14:41:39.504045 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.504031 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle\") pod \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " Jul 15 14:41:39.504282 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.504145 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg7xm\" (UniqueName: \"kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm\") pod \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\" (UID: \"7fe2e190-a4f2-4158-89d6-0c85c11f3d41\") " Jul 15 14:41:39.504746 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.504690 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle" (OuterVolumeSpecName: "bundle") pod "7fe2e190-a4f2-4158-89d6-0c85c11f3d41" (UID: "7fe2e190-a4f2-4158-89d6-0c85c11f3d41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:39.504921 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.504895 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle" (OuterVolumeSpecName: "bundle") pod "aac858ac-67c1-4bc2-bae5-cf5515a5ac73" (UID: "aac858ac-67c1-4bc2-bae5-cf5515a5ac73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:39.506228 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.506195 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl" (OuterVolumeSpecName: "kube-api-access-nmqbl") pod "aac858ac-67c1-4bc2-bae5-cf5515a5ac73" (UID: "aac858ac-67c1-4bc2-bae5-cf5515a5ac73"). InnerVolumeSpecName "kube-api-access-nmqbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:41:39.506321 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.506292 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm" (OuterVolumeSpecName: "kube-api-access-dg7xm") pod "7fe2e190-a4f2-4158-89d6-0c85c11f3d41" (UID: "7fe2e190-a4f2-4158-89d6-0c85c11f3d41"). InnerVolumeSpecName "kube-api-access-dg7xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:41:39.513174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.513117 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util" (OuterVolumeSpecName: "util") pod "7fe2e190-a4f2-4158-89d6-0c85c11f3d41" (UID: "7fe2e190-a4f2-4158-89d6-0c85c11f3d41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:39.518825 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.518807 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd5f58986-x7d6p"] Jul 15 14:41:39.519117 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519104 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="util" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519118 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="util" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519126 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="pull" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519131 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="pull" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519139 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="pull" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519144 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="pull" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519158 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="util" Jul 15 14:41:39.519162 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519163 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="util" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519169 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519174 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519182 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519186 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="util" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519199 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="util" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519205 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="pull" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519209 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="pull" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519214 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519219 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519283 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac858ac-67c1-4bc2-bae5-cf5515a5ac73" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519292 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61340d45-9b18-4baf-8ded-667e3bf32e08" containerName="extract" Jul 15 14:41:39.519348 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.519301 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe2e190-a4f2-4158-89d6-0c85c11f3d41" containerName="extract" Jul 15 14:41:39.530860 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.530843 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.550712 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.550693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5f58986-x7d6p"] Jul 15 14:41:39.604783 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-oauth-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.604783 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.604937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-service-ca\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.604937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-oauth-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.604937 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdxx\" (UniqueName: \"kubernetes.io/projected/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-kube-api-access-zxdxx\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.604977 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-trusted-ca-bundle\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.605032 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmqbl\" (UniqueName: \"kubernetes.io/projected/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-kube-api-access-nmqbl\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.605051 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.605084 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.605091 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.605122 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.605100 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg7xm\" (UniqueName: \"kubernetes.io/projected/7fe2e190-a4f2-4158-89d6-0c85c11f3d41-kube-api-access-dg7xm\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.610771 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.610731 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util" (OuterVolumeSpecName: "util") pod "aac858ac-67c1-4bc2-bae5-cf5515a5ac73" (UID: "aac858ac-67c1-4bc2-bae5-cf5515a5ac73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:39.706261 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-oauth-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706332 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706332 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-service-ca\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-oauth-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdxx\" (UniqueName: \"kubernetes.io/projected/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-kube-api-access-zxdxx\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706514 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706514 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-trusted-ca-bundle\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.706514 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.706493 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aac858ac-67c1-4bc2-bae5-cf5515a5ac73-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:39.707046 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.707025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-oauth-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.707237 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.707215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.707284 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.707234 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-service-ca\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.707421 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.707407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-trusted-ca-bundle\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.708649 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.708631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-oauth-config\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.708723 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.708709 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-console-serving-cert\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.724588 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.724562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdxx\" (UniqueName: \"kubernetes.io/projected/809d3aa7-81e9-4ea3-bc04-a7b32fbb7252-kube-api-access-zxdxx\") pod \"console-fd5f58986-x7d6p\" (UID: \"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252\") " pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.838740 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.838715 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:39.981995 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:39.981976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5f58986-x7d6p"] Jul 15 14:41:39.984045 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:39.983998 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809d3aa7_81e9_4ea3_bc04_a7b32fbb7252.slice/crio-4df802de7c23dcdc2d9c482f2d0f62bd23ec1546db5649329ae221968e79993d WatchSource:0}: Error finding container 4df802de7c23dcdc2d9c482f2d0f62bd23ec1546db5649329ae221968e79993d: Status 404 returned error can't find the container with id 4df802de7c23dcdc2d9c482f2d0f62bd23ec1546db5649329ae221968e79993d Jul 15 14:41:40.269069 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.269043 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" Jul 15 14:41:40.269373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.269074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl" event={"ID":"7fe2e190-a4f2-4158-89d6-0c85c11f3d41","Type":"ContainerDied","Data":"e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d"} Jul 15 14:41:40.269373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.269118 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50d73b8ef743f6ba2f4f84c402b6fd0938fae26c32ca58f1ff77f717c69065d" Jul 15 14:41:40.271132 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.271094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" event={"ID":"aac858ac-67c1-4bc2-bae5-cf5515a5ac73","Type":"ContainerDied","Data":"0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e"} Jul 15 14:41:40.271132 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.271124 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn" Jul 15 14:41:40.271313 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.271149 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0a2bf5594598afac136b27e7f35a05a439af1d540cc373fbaf23f0cfbc024e" Jul 15 14:41:40.272697 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.272668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5f58986-x7d6p" event={"ID":"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252","Type":"ContainerStarted","Data":"992bcc9a0ec7b4d24bd37e2d284cacc49097a28962a9564a797699132c5fbf0a"} Jul 15 14:41:40.272781 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.272713 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5f58986-x7d6p" event={"ID":"809d3aa7-81e9-4ea3-bc04-a7b32fbb7252","Type":"ContainerStarted","Data":"4df802de7c23dcdc2d9c482f2d0f62bd23ec1546db5649329ae221968e79993d"} Jul 15 14:41:40.314918 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.314873 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd5f58986-x7d6p" podStartSLOduration=1.314860001 podStartE2EDuration="1.314860001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:41:40.312089011 +0000 UTC m=+470.721708283" watchObservedRunningTime="2026-07-15 14:41:40.314860001 +0000 UTC m=+470.724479345" Jul 15 14:41:40.404552 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.404531 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:40.513724 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.513671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74wl9\" (UniqueName: \"kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9\") pod \"834e72c5-f717-464f-ab98-94b84775339e\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " Jul 15 14:41:40.513724 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.513716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle\") pod \"834e72c5-f717-464f-ab98-94b84775339e\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " Jul 15 14:41:40.513864 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.513757 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util\") pod \"834e72c5-f717-464f-ab98-94b84775339e\" (UID: \"834e72c5-f717-464f-ab98-94b84775339e\") " Jul 15 14:41:40.514334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.514307 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle" (OuterVolumeSpecName: "bundle") pod "834e72c5-f717-464f-ab98-94b84775339e" (UID: "834e72c5-f717-464f-ab98-94b84775339e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:40.515689 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.515664 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9" (OuterVolumeSpecName: "kube-api-access-74wl9") pod "834e72c5-f717-464f-ab98-94b84775339e" (UID: "834e72c5-f717-464f-ab98-94b84775339e"). InnerVolumeSpecName "kube-api-access-74wl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:41:40.518444 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.518422 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util" (OuterVolumeSpecName: "util") pod "834e72c5-f717-464f-ab98-94b84775339e" (UID: "834e72c5-f717-464f-ab98-94b84775339e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:41:40.614802 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.614780 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:40.614802 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.614799 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/834e72c5-f717-464f-ab98-94b84775339e-util\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:40.614939 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:40.614808 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wl9\" (UniqueName: \"kubernetes.io/projected/834e72c5-f717-464f-ab98-94b84775339e-kube-api-access-74wl9\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:41:41.279745 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:41.279712 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" Jul 15 14:41:41.279745 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:41.279743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5" event={"ID":"834e72c5-f717-464f-ab98-94b84775339e","Type":"ContainerDied","Data":"c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e"} Jul 15 14:41:41.280217 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:41.279784 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f5bb7cf38af49f5b0918034e4b1d7203a7954beeef502c9b615d143c29980e" Jul 15 14:41:49.839482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:49.839438 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:49.839911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:49.839556 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:49.843976 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:49.843940 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:50.327226 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:50.327203 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd5f58986-x7d6p" Jul 15 14:41:50.430599 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:50.430567 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:41:58.736582 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736553 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n"] Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736870 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="pull" Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736885 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="pull" Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736900 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="util" Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736907 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="util" Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736920 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="extract" Jul 15 14:41:58.736947 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736926 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="extract" Jul 15 14:41:58.737141 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.736983 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="834e72c5-f717-464f-ab98-94b84775339e" containerName="extract" Jul 15 14:41:58.739934 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.739919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.742202 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.742179 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 14:41:58.742332 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.742200 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7wm4v\"" Jul 15 14:41:58.743021 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.743001 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 14:41:58.752278 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.752257 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n"] Jul 15 14:41:58.846772 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.846750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqtv\" (UniqueName: \"kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.846860 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.846783 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.947868 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.947843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqtv\" (UniqueName: \"kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.947958 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.947877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.948229 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.948213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:58.971112 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:58.971086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqtv\" (UniqueName: \"kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:59.049641 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:59.049625 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:41:59.205138 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:41:59.205086 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5216400a_b699_4fef_aa61_8eb53c53189b.slice/crio-3adb8d6777056041743860711fb8c6362c793f423385e27f4aa70de7e4f1dd6f WatchSource:0}: Error finding container 3adb8d6777056041743860711fb8c6362c793f423385e27f4aa70de7e4f1dd6f: Status 404 returned error can't find the container with id 3adb8d6777056041743860711fb8c6362c793f423385e27f4aa70de7e4f1dd6f Jul 15 14:41:59.206607 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:59.206581 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n"] Jul 15 14:41:59.369291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:41:59.369215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" event={"ID":"5216400a-b699-4fef-aa61-8eb53c53189b","Type":"ContainerStarted","Data":"3adb8d6777056041743860711fb8c6362c793f423385e27f4aa70de7e4f1dd6f"} Jul 15 14:42:08.413469 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:08.413436 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" event={"ID":"5216400a-b699-4fef-aa61-8eb53c53189b","Type":"ContainerStarted","Data":"4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61"} Jul 15 14:42:08.413809 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:08.413582 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:08.443828 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:08.443787 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" podStartSLOduration=1.447673075 podStartE2EDuration="10.443773567s" totalImagesPullingTime="8.996100492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:08.442473392 +0000 UTC m=+498.852092670" watchObservedRunningTime="2026-07-15 14:42:08.443773567 +0000 UTC m=+498.853392846" Jul 15 14:42:15.451018 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.450956 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b9cbff45d-4fnh2" podUID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" containerName="console" containerID="cri-o://c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3" gracePeriod=15 Jul 15 14:42:15.689935 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.689913 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:42:15.778737 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778683 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778737 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778797 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778813 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778828 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcp4k\" (UniqueName: \"kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.778911 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.778842 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca\") pod \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\" (UID: \"f8ec1975-5b9b-48bf-8b51-51d1672a797e\") " Jul 15 14:42:15.779181 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.779029 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:42:15.779245 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.779223 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config" (OuterVolumeSpecName: "console-config") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:42:15.779313 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.779292 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:42:15.779393 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.779379 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:42:15.780884 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.780860 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:42:15.781373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.781347 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:42:15.781463 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.781390 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k" (OuterVolumeSpecName: "kube-api-access-wcp4k") pod "f8ec1975-5b9b-48bf-8b51-51d1672a797e" (UID: "f8ec1975-5b9b-48bf-8b51-51d1672a797e"). InnerVolumeSpecName "kube-api-access-wcp4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:42:15.879921 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879901 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.879921 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879921 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.880291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879930 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8ec1975-5b9b-48bf-8b51-51d1672a797e-console-oauth-config\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.880291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879940 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcp4k\" (UniqueName: \"kubernetes.io/projected/f8ec1975-5b9b-48bf-8b51-51d1672a797e-kube-api-access-wcp4k\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.880291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879949 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.880291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879956 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-trusted-ca-bundle\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:15.880291 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:15.879973 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8ec1975-5b9b-48bf-8b51-51d1672a797e-oauth-serving-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:16.454613 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.454543 2567 generic.go:356] "Generic (PLEG): container finished" podID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" containerID="c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3" exitCode=0 Jul 15 14:42:16.454613 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.454610 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cbff45d-4fnh2" Jul 15 14:42:16.455084 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.454639 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cbff45d-4fnh2" event={"ID":"f8ec1975-5b9b-48bf-8b51-51d1672a797e","Type":"ContainerDied","Data":"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3"} Jul 15 14:42:16.455084 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.454739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cbff45d-4fnh2" event={"ID":"f8ec1975-5b9b-48bf-8b51-51d1672a797e","Type":"ContainerDied","Data":"16675b983ce29b0fdea94cf518543a64f5bff17bacb0e132ea75a723c02fdaab"} Jul 15 14:42:16.455084 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.454769 2567 scope.go:117] "RemoveContainer" containerID="c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3" Jul 15 14:42:16.463851 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.463833 2567 scope.go:117] "RemoveContainer" containerID="c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3" Jul 15 14:42:16.464148 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:16.464127 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3\": container with ID starting with c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3 not found: ID does not exist" containerID="c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3" Jul 15 14:42:16.464212 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.464154 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3"} err="failed to get container status \"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3\": rpc error: code = NotFound desc = could not find container \"c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3\": container with ID starting with c490a1041fb0e35339550faec38e0f4c7a4ef4ae66d17d676f87de2acb73feb3 not found: ID does not exist" Jul 15 14:42:16.476239 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.476216 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:42:16.483710 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:16.483691 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b9cbff45d-4fnh2"] Jul 15 14:42:18.165664 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:18.165631 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" path="/var/lib/kubelet/pods/f8ec1975-5b9b-48bf-8b51-51d1672a797e/volumes" Jul 15 14:42:19.420221 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:19.420193 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:21.649339 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.649296 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n"] Jul 15 14:42:21.649748 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.649547 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" containerName="manager" containerID="cri-o://4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61" gracePeriod=2 Jul 15 14:42:21.651435 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:21.651405 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" is forbidden: User \"system:node:ip-10-0-141-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-12.ec2.internal' and this object" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:21.651664 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.651642 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n"] Jul 15 14:42:21.676330 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.676294 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:21.676942 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.676917 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" containerName="manager" Jul 15 14:42:21.676942 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.676938 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" containerName="manager" Jul 15 14:42:21.677048 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.676949 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" containerName="console" Jul 15 14:42:21.677048 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.676954 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" containerName="console" Jul 15 14:42:21.677048 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.677012 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" containerName="manager" Jul 15 14:42:21.677048 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.677020 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ec1975-5b9b-48bf-8b51-51d1672a797e" containerName="console" Jul 15 14:42:21.678961 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.678941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.699708 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.699679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:21.718334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.718309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlq49\" (UniqueName: \"kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.718603 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.718579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.719086 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:21.719046 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" is forbidden: User \"system:node:ip-10-0-141-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-12.ec2.internal' and this object" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:21.819879 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.819842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.820103 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.819895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlq49\" (UniqueName: \"kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.820283 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.820259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.831146 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.831123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlq49\" (UniqueName: \"kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8cbx\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:21.893986 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.893968 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:21.896400 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:21.896376 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xst6n\" is forbidden: User \"system:node:ip-10-0-141-12.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-12.ec2.internal' and this object" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:21.920955 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.920906 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume\") pod \"5216400a-b699-4fef-aa61-8eb53c53189b\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " Jul 15 14:42:21.921043 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.920957 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlqtv\" (UniqueName: \"kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv\") pod \"5216400a-b699-4fef-aa61-8eb53c53189b\" (UID: \"5216400a-b699-4fef-aa61-8eb53c53189b\") " Jul 15 14:42:21.921369 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.921347 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5216400a-b699-4fef-aa61-8eb53c53189b" (UID: "5216400a-b699-4fef-aa61-8eb53c53189b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:42:21.922830 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:21.922812 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv" (OuterVolumeSpecName: "kube-api-access-rlqtv") pod "5216400a-b699-4fef-aa61-8eb53c53189b" (UID: "5216400a-b699-4fef-aa61-8eb53c53189b"). InnerVolumeSpecName "kube-api-access-rlqtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:42:22.021724 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.021696 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:22.021926 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.021907 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqtv\" (UniqueName: \"kubernetes.io/projected/5216400a-b699-4fef-aa61-8eb53c53189b-kube-api-access-rlqtv\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:22.021974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.021928 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5216400a-b699-4fef-aa61-8eb53c53189b-extensions-socket-volume\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:22.164707 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.164676 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5216400a-b699-4fef-aa61-8eb53c53189b" path="/var/lib/kubelet/pods/5216400a-b699-4fef-aa61-8eb53c53189b/volumes" Jul 15 14:42:22.225172 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.225144 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:22.227139 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:42:22.227118 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2507d23_20d0_4182_b252_0c046b9fe779.slice/crio-1654f333dcdb809cab815de97e8b08e39350bac29cab739ed2783988980a9db0 WatchSource:0}: Error finding container 1654f333dcdb809cab815de97e8b08e39350bac29cab739ed2783988980a9db0: Status 404 returned error can't find the container with id 1654f333dcdb809cab815de97e8b08e39350bac29cab739ed2783988980a9db0 Jul 15 14:42:22.488508 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.488446 2567 generic.go:356] "Generic (PLEG): container finished" podID="5216400a-b699-4fef-aa61-8eb53c53189b" containerID="4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61" exitCode=0 Jul 15 14:42:22.488508 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.488492 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xst6n" Jul 15 14:42:22.488679 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.488527 2567 scope.go:117] "RemoveContainer" containerID="4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61" Jul 15 14:42:22.490707 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.490587 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" event={"ID":"b2507d23-20d0-4182-b252-0c046b9fe779","Type":"ContainerStarted","Data":"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a"} Jul 15 14:42:22.490707 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.490646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" event={"ID":"b2507d23-20d0-4182-b252-0c046b9fe779","Type":"ContainerStarted","Data":"1654f333dcdb809cab815de97e8b08e39350bac29cab739ed2783988980a9db0"} Jul 15 14:42:22.490865 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.490799 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:22.497356 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.497337 2567 scope.go:117] "RemoveContainer" containerID="4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61" Jul 15 14:42:22.497595 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:22.497579 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61\": container with ID starting with 4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61 not found: ID does not exist" containerID="4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61" Jul 15 14:42:22.497653 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.497608 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61"} err="failed to get container status \"4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61\": rpc error: code = NotFound desc = could not find container \"4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61\": container with ID starting with 4945d1ad3361b8e2f38e84672f25db393eadc9490aa3ec2e92dcf64645319c61 not found: ID does not exist" Jul 15 14:42:22.527437 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:22.527382 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" podStartSLOduration=1.5273660919999998 podStartE2EDuration="1.527366092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:22.523600443 +0000 UTC m=+512.933219722" watchObservedRunningTime="2026-07-15 14:42:22.527366092 +0000 UTC m=+512.936985372" Jul 15 14:42:33.499204 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.499173 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:33.877143 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.877118 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:42:33.880567 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.880550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:33.907389 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.907362 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:42:33.924141 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.924121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt47\" (UniqueName: \"kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:33.924248 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:33.924182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.025364 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.025341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.025475 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.025400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt47\" (UniqueName: \"kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.025691 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.025672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.049137 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.049115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt47\" (UniqueName: \"kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47\") pod \"kuadrant-operator-controller-manager-86878c769f-g4zss\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.190456 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.190403 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.339378 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.339323 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:42:34.341778 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:42:34.341752 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1fe44b_0da2_4384_adad_9cb79a223cb7.slice/crio-e5a173c6c5a40a2204a43f6df716baf3a51d23710dccb6ffa9280f6c977ea123 WatchSource:0}: Error finding container e5a173c6c5a40a2204a43f6df716baf3a51d23710dccb6ffa9280f6c977ea123: Status 404 returned error can't find the container with id e5a173c6c5a40a2204a43f6df716baf3a51d23710dccb6ffa9280f6c977ea123 Jul 15 14:42:34.560567 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.560533 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" event={"ID":"7b1fe44b-0da2-4384-adad-9cb79a223cb7","Type":"ContainerStarted","Data":"323a531942d3dc609e9580182103535e2d2e708edbf60692228e115b8423e933"} Jul 15 14:42:34.560932 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.560605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" event={"ID":"7b1fe44b-0da2-4384-adad-9cb79a223cb7","Type":"ContainerStarted","Data":"e5a173c6c5a40a2204a43f6df716baf3a51d23710dccb6ffa9280f6c977ea123"} Jul 15 14:42:34.560932 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.560639 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:34.607626 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:34.607577 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" podStartSLOduration=1.607563401 podStartE2EDuration="1.607563401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:34.606472476 +0000 UTC m=+525.016091756" watchObservedRunningTime="2026-07-15 14:42:34.607563401 +0000 UTC m=+525.017182678" Jul 15 14:42:45.567470 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:45.567382 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:42:45.649343 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:45.649317 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:45.649511 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:45.649489 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" podUID="b2507d23-20d0-4182-b252-0c046b9fe779" containerName="manager" containerID="cri-o://8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a" gracePeriod=10 Jul 15 14:42:45.889744 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:45.889720 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:46.024461 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.024438 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume\") pod \"b2507d23-20d0-4182-b252-0c046b9fe779\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " Jul 15 14:42:46.024601 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.024483 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlq49\" (UniqueName: \"kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49\") pod \"b2507d23-20d0-4182-b252-0c046b9fe779\" (UID: \"b2507d23-20d0-4182-b252-0c046b9fe779\") " Jul 15 14:42:46.024822 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.024799 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b2507d23-20d0-4182-b252-0c046b9fe779" (UID: "b2507d23-20d0-4182-b252-0c046b9fe779"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:42:46.026423 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.026389 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49" (OuterVolumeSpecName: "kube-api-access-jlq49") pod "b2507d23-20d0-4182-b252-0c046b9fe779" (UID: "b2507d23-20d0-4182-b252-0c046b9fe779"). InnerVolumeSpecName "kube-api-access-jlq49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:42:46.125712 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.125661 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2507d23-20d0-4182-b252-0c046b9fe779-extensions-socket-volume\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:46.125712 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.125681 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlq49\" (UniqueName: \"kubernetes.io/projected/b2507d23-20d0-4182-b252-0c046b9fe779-kube-api-access-jlq49\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:42:46.623810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.623778 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2507d23-20d0-4182-b252-0c046b9fe779" containerID="8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a" exitCode=0 Jul 15 14:42:46.624310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.623820 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" Jul 15 14:42:46.624310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.623862 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" event={"ID":"b2507d23-20d0-4182-b252-0c046b9fe779","Type":"ContainerDied","Data":"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a"} Jul 15 14:42:46.624310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.623913 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx" event={"ID":"b2507d23-20d0-4182-b252-0c046b9fe779","Type":"ContainerDied","Data":"1654f333dcdb809cab815de97e8b08e39350bac29cab739ed2783988980a9db0"} Jul 15 14:42:46.624310 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.623954 2567 scope.go:117] "RemoveContainer" containerID="8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a" Jul 15 14:42:46.632838 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.632817 2567 scope.go:117] "RemoveContainer" containerID="8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a" Jul 15 14:42:46.633131 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:42:46.633112 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a\": container with ID starting with 8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a not found: ID does not exist" containerID="8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a" Jul 15 14:42:46.633184 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.633140 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a"} err="failed to get container status \"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a\": rpc error: code = NotFound desc = could not find container \"8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a\": container with ID starting with 8fe362e2a26051968050b1bbe085498da4dd8159e512db868cd4e403fc7d920a not found: ID does not exist" Jul 15 14:42:46.663776 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.663754 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:46.687285 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:46.687262 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8cbx"] Jul 15 14:42:48.164876 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:42:48.164842 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2507d23-20d0-4182-b252-0c046b9fe779" path="/var/lib/kubelet/pods/b2507d23-20d0-4182-b252-0c046b9fe779/volumes" Jul 15 14:43:07.286373 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.286333 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:07.286703 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.286669 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2507d23-20d0-4182-b252-0c046b9fe779" containerName="manager" Jul 15 14:43:07.286703 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.286680 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2507d23-20d0-4182-b252-0c046b9fe779" containerName="manager" Jul 15 14:43:07.286780 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.286727 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2507d23-20d0-4182-b252-0c046b9fe779" containerName="manager" Jul 15 14:43:07.289551 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.289536 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.291684 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.291662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hxgsr\"" Jul 15 14:43:07.292143 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.292126 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 14:43:07.301621 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.301601 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:07.373683 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.373655 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:07.376335 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.376306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.376466 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.376357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrzw\" (UniqueName: \"kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.477701 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.477679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.477809 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.477723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrzw\" (UniqueName: \"kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.478329 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.478311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.495602 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.495577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrzw\" (UniqueName: \"kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw\") pod \"limitador-limitador-64c94fd64f-kkfrn\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.599709 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.599686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:07.734156 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:07.734119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:07.735721 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:43:07.735669 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6a577c_9fab_4e6f_bde7_c16e1c1d7e6e.slice/crio-10e49a742004b9bf8bd439b7f5509197ba84f04e91c3cf41e64e624d076d3da5 WatchSource:0}: Error finding container 10e49a742004b9bf8bd439b7f5509197ba84f04e91c3cf41e64e624d076d3da5: Status 404 returned error can't find the container with id 10e49a742004b9bf8bd439b7f5509197ba84f04e91c3cf41e64e624d076d3da5 Jul 15 14:43:08.742472 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:08.742429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" event={"ID":"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e","Type":"ContainerStarted","Data":"10e49a742004b9bf8bd439b7f5509197ba84f04e91c3cf41e64e624d076d3da5"} Jul 15 14:43:10.754579 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:10.754548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" event={"ID":"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e","Type":"ContainerStarted","Data":"38ad239087e8e79e303e033fb612fd765a9126653e11b9b4ca03aa5849d502b1"} Jul 15 14:43:10.754894 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:10.754627 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:10.778273 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:10.778231 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" podStartSLOduration=0.894246642 podStartE2EDuration="3.778217235s" totalImagesPullingTime="2.883970593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:10.776582385 +0000 UTC m=+561.186201666" watchObservedRunningTime="2026-07-15 14:43:10.778217235 +0000 UTC m=+561.187836539" Jul 15 14:43:21.761085 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:21.761032 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:22.402220 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:22.402186 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:22.402406 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:22.402370 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" podUID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" containerName="limitador" containerID="cri-o://38ad239087e8e79e303e033fb612fd765a9126653e11b9b4ca03aa5849d502b1" gracePeriod=30 Jul 15 14:43:22.816793 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:22.816759 2567 generic.go:356] "Generic (PLEG): container finished" podID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" containerID="38ad239087e8e79e303e033fb612fd765a9126653e11b9b4ca03aa5849d502b1" exitCode=0 Jul 15 14:43:22.817193 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:22.816835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" event={"ID":"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e","Type":"ContainerDied","Data":"38ad239087e8e79e303e033fb612fd765a9126653e11b9b4ca03aa5849d502b1"} Jul 15 14:43:23.339240 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.339213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:23.509248 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.509144 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrzw\" (UniqueName: \"kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw\") pod \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " Jul 15 14:43:23.509248 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.509242 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file\") pod \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\" (UID: \"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e\") " Jul 15 14:43:23.509592 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.509562 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file" (OuterVolumeSpecName: "config-file") pod "aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" (UID: "aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:43:23.511232 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.511202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw" (OuterVolumeSpecName: "kube-api-access-xjrzw") pod "aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" (UID: "aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e"). InnerVolumeSpecName "kube-api-access-xjrzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:43:23.610735 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.610701 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-config-file\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:43:23.610735 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.610728 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrzw\" (UniqueName: \"kubernetes.io/projected/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e-kube-api-access-xjrzw\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:43:23.822829 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.822798 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" event={"ID":"aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e","Type":"ContainerDied","Data":"10e49a742004b9bf8bd439b7f5509197ba84f04e91c3cf41e64e624d076d3da5"} Jul 15 14:43:23.823283 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.822859 2567 scope.go:117] "RemoveContainer" containerID="38ad239087e8e79e303e033fb612fd765a9126653e11b9b4ca03aa5849d502b1" Jul 15 14:43:23.823283 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.822809 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kkfrn" Jul 15 14:43:23.846214 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.846188 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:23.853814 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:23.853791 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kkfrn"] Jul 15 14:43:24.165051 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.164972 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" path="/var/lib/kubelet/pods/aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e/volumes" Jul 15 14:43:24.723735 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.723701 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qklxh"] Jul 15 14:43:24.724092 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.724077 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" containerName="limitador" Jul 15 14:43:24.724092 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.724093 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" containerName="limitador" Jul 15 14:43:24.724251 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.724178 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6a577c-9fab-4e6f-bde7-c16e1c1d7e6e" containerName="limitador" Jul 15 14:43:24.726893 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.726867 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:24.729193 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.729173 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xqqsh\"" Jul 15 14:43:24.729320 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.729218 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 14:43:24.729583 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.729565 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 14:43:24.729678 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.729587 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 14:43:24.742925 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.742900 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qklxh"] Jul 15 14:43:24.919755 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.919719 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59b80307-6fed-4352-83ba-4c6c959331f7-data\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:24.920174 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:24.919776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsjt\" (UniqueName: \"kubernetes.io/projected/59b80307-6fed-4352-83ba-4c6c959331f7-kube-api-access-bhsjt\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.020484 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.020381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsjt\" (UniqueName: \"kubernetes.io/projected/59b80307-6fed-4352-83ba-4c6c959331f7-kube-api-access-bhsjt\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.020484 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.020476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59b80307-6fed-4352-83ba-4c6c959331f7-data\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.020810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.020790 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59b80307-6fed-4352-83ba-4c6c959331f7-data\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.032007 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.031967 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsjt\" (UniqueName: \"kubernetes.io/projected/59b80307-6fed-4352-83ba-4c6c959331f7-kube-api-access-bhsjt\") pod \"postgres-69fb4594cf-qklxh\" (UID: \"59b80307-6fed-4352-83ba-4c6c959331f7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.037808 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.037786 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:25.179028 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:43:25.178995 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b80307_6fed_4352_83ba_4c6c959331f7.slice/crio-c897477db2abe03f9b58149f0ca89837d51795b0d44375518aa2b547dab8e7e0 WatchSource:0}: Error finding container c897477db2abe03f9b58149f0ca89837d51795b0d44375518aa2b547dab8e7e0: Status 404 returned error can't find the container with id c897477db2abe03f9b58149f0ca89837d51795b0d44375518aa2b547dab8e7e0 Jul 15 14:43:25.181214 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.181153 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qklxh"] Jul 15 14:43:25.837079 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:25.836976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" event={"ID":"59b80307-6fed-4352-83ba-4c6c959331f7","Type":"ContainerStarted","Data":"c897477db2abe03f9b58149f0ca89837d51795b0d44375518aa2b547dab8e7e0"} Jul 15 14:43:29.866336 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:29.866301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" event={"ID":"59b80307-6fed-4352-83ba-4c6c959331f7","Type":"ContainerStarted","Data":"28a2baad89f19e004250826e35eff8a8993dfdf575093e3d826c9540e23563d6"} Jul 15 14:43:29.866710 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:29.866408 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:29.887817 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:29.887770 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" podStartSLOduration=1.351986571 podStartE2EDuration="5.887754441s" totalImagesPullingTime="4.53576787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:29.886558634 +0000 UTC m=+580.296177913" watchObservedRunningTime="2026-07-15 14:43:29.887754441 +0000 UTC m=+580.297373718" Jul 15 14:43:35.900493 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:35.900464 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qklxh" Jul 15 14:43:46.043628 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.043595 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:43:46.047419 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.047386 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.050164 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.050138 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8b6p8\"" Jul 15 14:43:46.051321 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.051297 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 14:43:46.052087 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.052039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 14:43:46.057595 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.057574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:43:46.109153 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.109111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqdn\" (UniqueName: \"kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.109338 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.109172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.210674 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.210635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqdn\" (UniqueName: \"kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.210855 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.210721 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.213149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.213117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.220237 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.220208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqdn\" (UniqueName: \"kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn\") pod \"maas-controller-86dbf557b7-jkt7k\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.280607 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.280575 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:43:46.283033 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.283018 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.294204 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.294139 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:43:46.360533 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.360501 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:46.412186 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.412151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7vv\" (UniqueName: \"kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.412328 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.412213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.513101 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.513052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7vv\" (UniqueName: \"kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.513284 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.513122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.515576 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.515551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:43:46.515698 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.515648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.516926 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:43:46.516903 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e579fa_0e1d_4936_8eec_55b7f61620fb.slice/crio-37a43a4272e051ce4d072f7b6954836802ae28909bdeb3d66d021943fc3a1489 WatchSource:0}: Error finding container 37a43a4272e051ce4d072f7b6954836802ae28909bdeb3d66d021943fc3a1489: Status 404 returned error can't find the container with id 37a43a4272e051ce4d072f7b6954836802ae28909bdeb3d66d021943fc3a1489 Jul 15 14:43:46.526149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.526128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7vv\" (UniqueName: \"kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv\") pod \"maas-controller-7565848f6c-m78z9\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.594845 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.594817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:46.729998 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.729973 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:43:46.731958 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:43:46.731921 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4b9053_8c98_420d_b24b_adaf1fa87271.slice/crio-ee73e22c97ef1e1f5e9b00430cc4e474c72514d31655b02e98264210b7bde49a WatchSource:0}: Error finding container ee73e22c97ef1e1f5e9b00430cc4e474c72514d31655b02e98264210b7bde49a: Status 404 returned error can't find the container with id ee73e22c97ef1e1f5e9b00430cc4e474c72514d31655b02e98264210b7bde49a Jul 15 14:43:46.951315 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.951227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" event={"ID":"46e579fa-0e1d-4936-8eec-55b7f61620fb","Type":"ContainerStarted","Data":"37a43a4272e051ce4d072f7b6954836802ae28909bdeb3d66d021943fc3a1489"} Jul 15 14:43:46.952342 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:46.952316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-m78z9" event={"ID":"db4b9053-8c98-420d-b24b-adaf1fa87271","Type":"ContainerStarted","Data":"ee73e22c97ef1e1f5e9b00430cc4e474c72514d31655b02e98264210b7bde49a"} Jul 15 14:43:49.976261 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:49.976224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-m78z9" event={"ID":"db4b9053-8c98-420d-b24b-adaf1fa87271","Type":"ContainerStarted","Data":"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d"} Jul 15 14:43:49.976673 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:49.976295 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:43:49.977568 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:49.977539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" event={"ID":"46e579fa-0e1d-4936-8eec-55b7f61620fb","Type":"ContainerStarted","Data":"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b"} Jul 15 14:43:49.977654 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:49.977579 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:43:49.998872 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:49.998821 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7565848f6c-m78z9" podStartSLOduration=1.3376119499999999 podStartE2EDuration="3.998806335s" totalImagesPullingTime="2.661194385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:49.998159763 +0000 UTC m=+600.407779040" watchObservedRunningTime="2026-07-15 14:43:49.998806335 +0000 UTC m=+600.408425612" Jul 15 14:43:50.023219 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:50.023170 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" podStartSLOduration=1.156157504 podStartE2EDuration="4.02315148s" totalImagesPullingTime="2.866993976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:50.022600299 +0000 UTC m=+600.432219578" watchObservedRunningTime="2026-07-15 14:43:50.02315148 +0000 UTC m=+600.432770758" Jul 15 14:43:50.026137 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:50.026117 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:43:50.026271 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:43:50.026228 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:44:00.984332 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:00.984295 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:44:00.985880 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:00.984399 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:44:01.046397 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.046361 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:44:01.046605 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.046576 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" podUID="46e579fa-0e1d-4936-8eec-55b7f61620fb" containerName="manager" containerID="cri-o://9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b" gracePeriod=30 Jul 15 14:44:01.286474 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.286448 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:44:01.450312 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.450279 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert\") pod \"46e579fa-0e1d-4936-8eec-55b7f61620fb\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " Jul 15 14:44:01.450507 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.450384 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqdn\" (UniqueName: \"kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn\") pod \"46e579fa-0e1d-4936-8eec-55b7f61620fb\" (UID: \"46e579fa-0e1d-4936-8eec-55b7f61620fb\") " Jul 15 14:44:01.452505 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.452463 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "46e579fa-0e1d-4936-8eec-55b7f61620fb" (UID: "46e579fa-0e1d-4936-8eec-55b7f61620fb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:44:01.452623 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.452537 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn" (OuterVolumeSpecName: "kube-api-access-4rqdn") pod "46e579fa-0e1d-4936-8eec-55b7f61620fb" (UID: "46e579fa-0e1d-4936-8eec-55b7f61620fb"). InnerVolumeSpecName "kube-api-access-4rqdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:44:01.551424 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.551388 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqdn\" (UniqueName: \"kubernetes.io/projected/46e579fa-0e1d-4936-8eec-55b7f61620fb-kube-api-access-4rqdn\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:44:01.551424 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:01.551424 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46e579fa-0e1d-4936-8eec-55b7f61620fb-webhook-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:44:02.037773 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.037691 2567 generic.go:356] "Generic (PLEG): container finished" podID="46e579fa-0e1d-4936-8eec-55b7f61620fb" containerID="9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b" exitCode=0 Jul 15 14:44:02.037773 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.037733 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" Jul 15 14:44:02.037773 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.037765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" event={"ID":"46e579fa-0e1d-4936-8eec-55b7f61620fb","Type":"ContainerDied","Data":"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b"} Jul 15 14:44:02.038270 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.037805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86dbf557b7-jkt7k" event={"ID":"46e579fa-0e1d-4936-8eec-55b7f61620fb","Type":"ContainerDied","Data":"37a43a4272e051ce4d072f7b6954836802ae28909bdeb3d66d021943fc3a1489"} Jul 15 14:44:02.038270 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.037830 2567 scope.go:117] "RemoveContainer" containerID="9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b" Jul 15 14:44:02.046441 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.046422 2567 scope.go:117] "RemoveContainer" containerID="9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b" Jul 15 14:44:02.046688 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:44:02.046668 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b\": container with ID starting with 9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b not found: ID does not exist" containerID="9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b" Jul 15 14:44:02.046748 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.046697 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b"} err="failed to get container status \"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b\": rpc error: code = NotFound desc = could not find container \"9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b\": container with ID starting with 9bdfc3560eab374643f7153ecc49327cd4da64ddc3605c7dde42fb3fca133d5b not found: ID does not exist" Jul 15 14:44:02.063019 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.062986 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:44:02.069791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.069766 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86dbf557b7-jkt7k"] Jul 15 14:44:02.164698 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:02.164665 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e579fa-0e1d-4936-8eec-55b7f61620fb" path="/var/lib/kubelet/pods/46e579fa-0e1d-4936-8eec-55b7f61620fb/volumes" Jul 15 14:44:22.211149 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.211112 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7966b86-dldrb"] Jul 15 14:44:22.211700 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.211599 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e579fa-0e1d-4936-8eec-55b7f61620fb" containerName="manager" Jul 15 14:44:22.211700 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.211617 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e579fa-0e1d-4936-8eec-55b7f61620fb" containerName="manager" Jul 15 14:44:22.211700 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.211700 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e579fa-0e1d-4936-8eec-55b7f61620fb" containerName="manager" Jul 15 14:44:22.217140 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.217119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.220349 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.220322 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-chv7n\"" Jul 15 14:44:22.220742 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.220716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlrj\" (UniqueName: \"kubernetes.io/projected/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-kube-api-access-bmlrj\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.220871 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.220775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-openshift-service-ca\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.220871 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.220857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.220979 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.220919 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 14:44:22.231109 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.231079 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7966b86-dldrb"] Jul 15 14:44:22.321648 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.321615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlrj\" (UniqueName: \"kubernetes.io/projected/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-kube-api-access-bmlrj\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.321826 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.321654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-openshift-service-ca\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.321826 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.321709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.321941 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:44:22.321902 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 15 14:44:22.321996 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:44:22.321979 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls podName:e4d00c76-b9bd-4145-9d39-ce79f1f628c7 nodeName:}" failed. No retries permitted until 2026-07-15 14:44:22.82196216 +0000 UTC m=+633.231581420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls") pod "maas-api-7966b86-dldrb" (UID: "e4d00c76-b9bd-4145-9d39-ce79f1f628c7") : secret "maas-api-serving-cert" not found Jul 15 14:44:22.322431 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.322407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-openshift-service-ca\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.333791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.333767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlrj\" (UniqueName: \"kubernetes.io/projected/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-kube-api-access-bmlrj\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.825891 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.825858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.828197 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.828177 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d00c76-b9bd-4145-9d39-ce79f1f628c7-maas-api-tls\") pod \"maas-api-7966b86-dldrb\" (UID: \"e4d00c76-b9bd-4145-9d39-ce79f1f628c7\") " pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.828299 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.828287 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:22.981935 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.981903 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7966b86-dldrb"] Jul 15 14:44:22.984431 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:44:22.984401 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d00c76_b9bd_4145_9d39_ce79f1f628c7.slice/crio-e6d92ab4bc139f8ed5c8265018033a6402b67c266f16d83cfe1e55fa5014506f WatchSource:0}: Error finding container e6d92ab4bc139f8ed5c8265018033a6402b67c266f16d83cfe1e55fa5014506f: Status 404 returned error can't find the container with id e6d92ab4bc139f8ed5c8265018033a6402b67c266f16d83cfe1e55fa5014506f Jul 15 14:44:22.985639 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:22.985623 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:44:23.143206 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.143123 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" event={"ID":"e4d00c76-b9bd-4145-9d39-ce79f1f628c7","Type":"ContainerStarted","Data":"e6d92ab4bc139f8ed5c8265018033a6402b67c266f16d83cfe1e55fa5014506f"} Jul 15 14:44:23.157543 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.157510 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5qw62"] Jul 15 14:44:23.160641 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.160621 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.169617 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.169598 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 14:44:23.169750 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.169599 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bz6q4\"" Jul 15 14:44:23.187752 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.187722 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5qw62"] Jul 15 14:44:23.229593 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.229562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57d03ab-fd6d-46a1-986b-c632b83c4abb-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.229995 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.229600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j944\" (UniqueName: \"kubernetes.io/projected/c57d03ab-fd6d-46a1-986b-c632b83c4abb-kube-api-access-4j944\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.241338 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.241309 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-nfb5w"] Jul 15 14:44:23.243560 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.243545 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.267537 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.267506 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-nfb5w"] Jul 15 14:44:23.330846 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.330804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57d03ab-fd6d-46a1-986b-c632b83c4abb-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.331004 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.330852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j944\" (UniqueName: \"kubernetes.io/projected/c57d03ab-fd6d-46a1-986b-c632b83c4abb-kube-api-access-4j944\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.331004 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.330928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311fa312-9e09-40df-a78e-6a8a3cd1227d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.331004 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.330972 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66jw\" (UniqueName: \"kubernetes.io/projected/311fa312-9e09-40df-a78e-6a8a3cd1227d-kube-api-access-x66jw\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.331580 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.331561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57d03ab-fd6d-46a1-986b-c632b83c4abb-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.342041 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.342014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j944\" (UniqueName: \"kubernetes.io/projected/c57d03ab-fd6d-46a1-986b-c632b83c4abb-kube-api-access-4j944\") pod \"payload-pre-processing-74584bb9f8-5qw62\" (UID: \"c57d03ab-fd6d-46a1-986b-c632b83c4abb\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.431810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.431680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311fa312-9e09-40df-a78e-6a8a3cd1227d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.431810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.431779 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66jw\" (UniqueName: \"kubernetes.io/projected/311fa312-9e09-40df-a78e-6a8a3cd1227d-kube-api-access-x66jw\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.432483 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.432460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311fa312-9e09-40df-a78e-6a8a3cd1227d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.442212 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.442181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66jw\" (UniqueName: \"kubernetes.io/projected/311fa312-9e09-40df-a78e-6a8a3cd1227d-kube-api-access-x66jw\") pod \"payload-processing-f9b8d9b9b-nfb5w\" (UID: \"311fa312-9e09-40df-a78e-6a8a3cd1227d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.472166 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.472122 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:23.559400 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.559365 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:23.613297 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.613270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5qw62"] Jul 15 14:44:23.614422 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:44:23.614389 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57d03ab_fd6d_46a1_986b_c632b83c4abb.slice/crio-ae02400cf92d8d2c13562c72356f9817dde659db49a034c14ced5f6641e1fd50 WatchSource:0}: Error finding container ae02400cf92d8d2c13562c72356f9817dde659db49a034c14ced5f6641e1fd50: Status 404 returned error can't find the container with id ae02400cf92d8d2c13562c72356f9817dde659db49a034c14ced5f6641e1fd50 Jul 15 14:44:23.733241 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:23.733151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-nfb5w"] Jul 15 14:44:23.736477 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:44:23.736441 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311fa312_9e09_40df_a78e_6a8a3cd1227d.slice/crio-948c45339de52abac245babe62908f801fec4e9feec75baf1a66ac02a4f89c40 WatchSource:0}: Error finding container 948c45339de52abac245babe62908f801fec4e9feec75baf1a66ac02a4f89c40: Status 404 returned error can't find the container with id 948c45339de52abac245babe62908f801fec4e9feec75baf1a66ac02a4f89c40 Jul 15 14:44:24.150467 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:24.150431 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" event={"ID":"311fa312-9e09-40df-a78e-6a8a3cd1227d","Type":"ContainerStarted","Data":"948c45339de52abac245babe62908f801fec4e9feec75baf1a66ac02a4f89c40"} Jul 15 14:44:24.151941 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:24.151913 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" event={"ID":"c57d03ab-fd6d-46a1-986b-c632b83c4abb","Type":"ContainerStarted","Data":"ae02400cf92d8d2c13562c72356f9817dde659db49a034c14ced5f6641e1fd50"} Jul 15 14:44:25.159040 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.159006 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" event={"ID":"e4d00c76-b9bd-4145-9d39-ce79f1f628c7","Type":"ContainerStarted","Data":"c9c67666182f87e48e7321edf6c4103a2182192fce432a40bd4e9aef136fa3eb"} Jul 15 14:44:25.159456 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.159099 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:25.160828 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.160762 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" event={"ID":"c57d03ab-fd6d-46a1-986b-c632b83c4abb","Type":"ContainerStarted","Data":"76bd501f8b2d8314bbe0adc296f04071b15bf0cf36e355e3ede30d03070f9d0b"} Jul 15 14:44:25.160828 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.160816 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:25.162407 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.162384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" event={"ID":"311fa312-9e09-40df-a78e-6a8a3cd1227d","Type":"ContainerStarted","Data":"f84dd6429c18097ffcea9ea50c44d4caddbcdd510099535e1fc967ae6724900e"} Jul 15 14:44:25.162547 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.162514 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:25.183513 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.183412 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" podStartSLOduration=1.617651483 podStartE2EDuration="3.183396953s" totalImagesPullingTime="1.56574547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:44:25.181383589 +0000 UTC m=+635.591002862" watchObservedRunningTime="2026-07-15 14:44:25.183396953 +0000 UTC m=+635.593016231" Jul 15 14:44:25.203959 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.203909 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" podStartSLOduration=0.835451186 podStartE2EDuration="2.203893224s" totalImagesPullingTime="1.368442038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:44:25.202243775 +0000 UTC m=+635.611863056" watchObservedRunningTime="2026-07-15 14:44:25.203893224 +0000 UTC m=+635.613512530" Jul 15 14:44:25.223638 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:25.223591 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" podStartSLOduration=0.974613753 podStartE2EDuration="2.223576963s" totalImagesPullingTime="1.24896321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:44:25.221422256 +0000 UTC m=+635.631041536" watchObservedRunningTime="2026-07-15 14:44:25.223576963 +0000 UTC m=+635.633196241" Jul 15 14:44:31.173073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:31.173027 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7966b86-dldrb" Jul 15 14:44:36.168974 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:36.168942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-nfb5w" Jul 15 14:44:36.169414 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:36.169016 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5qw62" Jul 15 14:44:53.417312 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.417278 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6"] Jul 15 14:44:53.421791 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.421768 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.424305 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.424284 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 14:44:53.424524 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.424511 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 15 14:44:53.424576 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.424560 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-68jzd\"" Jul 15 14:44:53.425075 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.425051 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 14:44:53.436503 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.436478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6"] Jul 15 14:44:53.500311 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.500482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.500482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d570ad-ae52-4704-8b84-ec78d5e6f866-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.500482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72ns\" (UniqueName: \"kubernetes.io/projected/41d570ad-ae52-4704-8b84-ec78d5e6f866-kube-api-access-d72ns\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.500482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.500482 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.500462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601086 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601261 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d570ad-ae52-4704-8b84-ec78d5e6f866-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601261 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72ns\" (UniqueName: \"kubernetes.io/projected/41d570ad-ae52-4704-8b84-ec78d5e6f866-kube-api-access-d72ns\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601377 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601377 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601479 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601479 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601680 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601653 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.601808 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.601789 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.603443 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.603421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d570ad-ae52-4704-8b84-ec78d5e6f866-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.603551 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.603536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d570ad-ae52-4704-8b84-ec78d5e6f866-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.612334 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.612313 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72ns\" (UniqueName: \"kubernetes.io/projected/41d570ad-ae52-4704-8b84-ec78d5e6f866-kube-api-access-d72ns\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z2cq6\" (UID: \"41d570ad-ae52-4704-8b84-ec78d5e6f866\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.732419 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.732328 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:44:53.872443 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:53.872419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6"] Jul 15 14:44:53.874826 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:44:53.874798 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d570ad_ae52_4704_8b84_ec78d5e6f866.slice/crio-386002070eacd4ec15ac6ef091ffd3e3f770a7c09b9fe80ddba3ebeb5e6d5eb1 WatchSource:0}: Error finding container 386002070eacd4ec15ac6ef091ffd3e3f770a7c09b9fe80ddba3ebeb5e6d5eb1: Status 404 returned error can't find the container with id 386002070eacd4ec15ac6ef091ffd3e3f770a7c09b9fe80ddba3ebeb5e6d5eb1 Jul 15 14:44:54.315089 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:54.315027 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" event={"ID":"41d570ad-ae52-4704-8b84-ec78d5e6f866","Type":"ContainerStarted","Data":"386002070eacd4ec15ac6ef091ffd3e3f770a7c09b9fe80ddba3ebeb5e6d5eb1"} Jul 15 14:44:55.321731 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:55.321692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" event={"ID":"41d570ad-ae52-4704-8b84-ec78d5e6f866","Type":"ContainerStarted","Data":"b34861a8132ef69cae9ca5fb97df39a51bff34da77425031cb66a7eed093d3e1"} Jul 15 14:44:55.350510 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:44:55.350457 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" podStartSLOduration=1.185335433 podStartE2EDuration="2.350440472s" totalImagesPullingTime="1.165105039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:44:55.348288653 +0000 UTC m=+665.757907941" watchObservedRunningTime="2026-07-15 14:44:55.350440472 +0000 UTC m=+665.760059750" Jul 15 14:45:03.732989 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:45:03.732947 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:45:03.733535 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:45:03.733025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:45:03.746536 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:45:03.746509 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:45:04.377971 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:45:04.377941 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z2cq6" Jul 15 14:47:50.167813 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.167782 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:47:50.168437 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.168029 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7565848f6c-m78z9" podUID="db4b9053-8c98-420d-b24b-adaf1fa87271" containerName="manager" containerID="cri-o://a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d" gracePeriod=30 Jul 15 14:47:50.425736 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.425669 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:47:50.535112 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.535081 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp7vv\" (UniqueName: \"kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv\") pod \"db4b9053-8c98-420d-b24b-adaf1fa87271\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " Jul 15 14:47:50.535317 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.535128 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert\") pod \"db4b9053-8c98-420d-b24b-adaf1fa87271\" (UID: \"db4b9053-8c98-420d-b24b-adaf1fa87271\") " Jul 15 14:47:50.537272 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.537234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "db4b9053-8c98-420d-b24b-adaf1fa87271" (UID: "db4b9053-8c98-420d-b24b-adaf1fa87271"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:47:50.537379 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.537294 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv" (OuterVolumeSpecName: "kube-api-access-zp7vv") pod "db4b9053-8c98-420d-b24b-adaf1fa87271" (UID: "db4b9053-8c98-420d-b24b-adaf1fa87271"). InnerVolumeSpecName "kube-api-access-zp7vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:47:50.635919 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.635870 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7vv\" (UniqueName: \"kubernetes.io/projected/db4b9053-8c98-420d-b24b-adaf1fa87271-kube-api-access-zp7vv\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:47:50.635919 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:50.635912 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4b9053-8c98-420d-b24b-adaf1fa87271-webhook-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:47:51.222154 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.222117 2567 generic.go:356] "Generic (PLEG): container finished" podID="db4b9053-8c98-420d-b24b-adaf1fa87271" containerID="a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d" exitCode=0 Jul 15 14:47:51.222561 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.222160 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-m78z9" Jul 15 14:47:51.222561 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.222195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-m78z9" event={"ID":"db4b9053-8c98-420d-b24b-adaf1fa87271","Type":"ContainerDied","Data":"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d"} Jul 15 14:47:51.222561 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.222243 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-m78z9" event={"ID":"db4b9053-8c98-420d-b24b-adaf1fa87271","Type":"ContainerDied","Data":"ee73e22c97ef1e1f5e9b00430cc4e474c72514d31655b02e98264210b7bde49a"} Jul 15 14:47:51.222561 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.222264 2567 scope.go:117] "RemoveContainer" containerID="a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d" Jul 15 14:47:51.233091 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.233043 2567 scope.go:117] "RemoveContainer" containerID="a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d" Jul 15 14:47:51.233422 ip-10-0-141-12 kubenswrapper[2567]: E0715 14:47:51.233401 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d\": container with ID starting with a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d not found: ID does not exist" containerID="a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d" Jul 15 14:47:51.233530 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.233427 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d"} err="failed to get container status \"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d\": rpc error: code = NotFound desc = could not find container \"a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d\": container with ID starting with a8013c68c93c4ec1ecaf0779375d4c5ace99bb6c2e2baff7f053ed949205669d not found: ID does not exist" Jul 15 14:47:51.246354 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.246329 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:47:51.251785 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:51.251759 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7565848f6c-m78z9"] Jul 15 14:47:52.027419 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.027385 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7565848f6c-77z2q"] Jul 15 14:47:52.027758 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.027746 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4b9053-8c98-420d-b24b-adaf1fa87271" containerName="manager" Jul 15 14:47:52.027806 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.027758 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4b9053-8c98-420d-b24b-adaf1fa87271" containerName="manager" Jul 15 14:47:52.027841 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.027808 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4b9053-8c98-420d-b24b-adaf1fa87271" containerName="manager" Jul 15 14:47:52.032305 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.032283 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.041073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.040784 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 14:47:52.041073 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.040803 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8b6p8\"" Jul 15 14:47:52.043022 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.042992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 14:47:52.045014 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.044992 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7565848f6c-77z2q"] Jul 15 14:47:52.148417 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.148375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ncd\" (UniqueName: \"kubernetes.io/projected/2faefc27-2adf-4719-ad54-988323933549-kube-api-access-h9ncd\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.148604 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.148495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faefc27-2adf-4719-ad54-988323933549-webhook-cert\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.164071 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.164024 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4b9053-8c98-420d-b24b-adaf1fa87271" path="/var/lib/kubelet/pods/db4b9053-8c98-420d-b24b-adaf1fa87271/volumes" Jul 15 14:47:52.249123 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.249085 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faefc27-2adf-4719-ad54-988323933549-webhook-cert\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.249637 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.249185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ncd\" (UniqueName: \"kubernetes.io/projected/2faefc27-2adf-4719-ad54-988323933549-kube-api-access-h9ncd\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.251598 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.251575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2faefc27-2adf-4719-ad54-988323933549-webhook-cert\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.257751 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.257725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ncd\" (UniqueName: \"kubernetes.io/projected/2faefc27-2adf-4719-ad54-988323933549-kube-api-access-h9ncd\") pod \"maas-controller-7565848f6c-77z2q\" (UID: \"2faefc27-2adf-4719-ad54-988323933549\") " pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.351229 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.351195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:52.494211 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:52.494169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7565848f6c-77z2q"] Jul 15 14:47:52.495474 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:47:52.495450 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faefc27_2adf_4719_ad54_988323933549.slice/crio-49f1d565edec1584f340a39ff1182e790ef3f1f171d465588f0d08581a8b1735 WatchSource:0}: Error finding container 49f1d565edec1584f340a39ff1182e790ef3f1f171d465588f0d08581a8b1735: Status 404 returned error can't find the container with id 49f1d565edec1584f340a39ff1182e790ef3f1f171d465588f0d08581a8b1735 Jul 15 14:47:53.234992 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:53.234952 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-77z2q" event={"ID":"2faefc27-2adf-4719-ad54-988323933549","Type":"ContainerStarted","Data":"49f1d565edec1584f340a39ff1182e790ef3f1f171d465588f0d08581a8b1735"} Jul 15 14:47:54.244776 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:54.244732 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7565848f6c-77z2q" event={"ID":"2faefc27-2adf-4719-ad54-988323933549","Type":"ContainerStarted","Data":"4bbf75c4c10e0942db2036a2bef2d13f28d09e8c8616d5c246a60b6ee66d5e91"} Jul 15 14:47:54.245263 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:54.244862 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:47:54.265817 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:47:54.265762 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7565848f6c-77z2q" podStartSLOduration=1.366514437 podStartE2EDuration="2.265744792s" totalImagesPullingTime="899.230355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:47:54.264256381 +0000 UTC m=+844.673875652" watchObservedRunningTime="2026-07-15 14:47:54.265744792 +0000 UTC m=+844.675364069" Jul 15 14:48:05.252667 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:48:05.252639 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7565848f6c-77z2q" Jul 15 14:48:50.059514 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:48:50.059426 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:48:50.063313 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:48:50.063293 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:52:49.999968 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:49.999935 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:52:50.000550 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:50.000210 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" podUID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" containerName="manager" containerID="cri-o://323a531942d3dc609e9580182103535e2d2e708edbf60692228e115b8423e933" gracePeriod=10 Jul 15 14:52:52.834399 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.834367 2567 generic.go:356] "Generic (PLEG): container finished" podID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" containerID="323a531942d3dc609e9580182103535e2d2e708edbf60692228e115b8423e933" exitCode=0 Jul 15 14:52:52.834785 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.834439 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" event={"ID":"7b1fe44b-0da2-4384-adad-9cb79a223cb7","Type":"ContainerDied","Data":"323a531942d3dc609e9580182103535e2d2e708edbf60692228e115b8423e933"} Jul 15 14:52:52.955594 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.955566 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:52:52.980119 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.979977 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlt47\" (UniqueName: \"kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47\") pod \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " Jul 15 14:52:52.980119 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.980049 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume\") pod \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\" (UID: \"7b1fe44b-0da2-4384-adad-9cb79a223cb7\") " Jul 15 14:52:52.980409 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.980379 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7b1fe44b-0da2-4384-adad-9cb79a223cb7" (UID: "7b1fe44b-0da2-4384-adad-9cb79a223cb7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:52:52.982719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:52.982681 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47" (OuterVolumeSpecName: "kube-api-access-tlt47") pod "7b1fe44b-0da2-4384-adad-9cb79a223cb7" (UID: "7b1fe44b-0da2-4384-adad-9cb79a223cb7"). InnerVolumeSpecName "kube-api-access-tlt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:52:53.080691 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.080660 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlt47\" (UniqueName: \"kubernetes.io/projected/7b1fe44b-0da2-4384-adad-9cb79a223cb7-kube-api-access-tlt47\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:52:53.080691 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.080687 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b1fe44b-0da2-4384-adad-9cb79a223cb7-extensions-socket-volume\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 14:52:53.840632 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.840599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" event={"ID":"7b1fe44b-0da2-4384-adad-9cb79a223cb7","Type":"ContainerDied","Data":"e5a173c6c5a40a2204a43f6df716baf3a51d23710dccb6ffa9280f6c977ea123"} Jul 15 14:52:53.840632 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.840628 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss" Jul 15 14:52:53.841147 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.840664 2567 scope.go:117] "RemoveContainer" containerID="323a531942d3dc609e9580182103535e2d2e708edbf60692228e115b8423e933" Jul 15 14:52:53.863560 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.863534 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:52:53.867467 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:53.867441 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g4zss"] Jul 15 14:52:54.164478 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:52:54.164387 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" path="/var/lib/kubelet/pods/7b1fe44b-0da2-4384-adad-9cb79a223cb7/volumes" Jul 15 14:53:50.093384 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:50.093353 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:53:50.098091 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:50.098038 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:53:51.222541 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.222504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f"] Jul 15 14:53:51.223003 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.222986 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" containerName="manager" Jul 15 14:53:51.223076 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.223005 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" containerName="manager" Jul 15 14:53:51.223124 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.223105 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1fe44b-0da2-4384-adad-9cb79a223cb7" containerName="manager" Jul 15 14:53:51.226925 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.226906 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.229253 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.229225 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7wm4v\"" Jul 15 14:53:51.229473 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.229454 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 14:53:51.229719 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.229704 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 14:53:51.241455 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.241429 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f"] Jul 15 14:53:51.290699 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.290668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8535db98-63f7-42ff-9774-4d864939faf7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.290901 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.290728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dr8v\" (UniqueName: \"kubernetes.io/projected/8535db98-63f7-42ff-9774-4d864939faf7-kube-api-access-7dr8v\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.392209 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.392144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8535db98-63f7-42ff-9774-4d864939faf7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.392422 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.392264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dr8v\" (UniqueName: \"kubernetes.io/projected/8535db98-63f7-42ff-9774-4d864939faf7-kube-api-access-7dr8v\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.392550 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.392528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8535db98-63f7-42ff-9774-4d864939faf7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.413852 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.413816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dr8v\" (UniqueName: \"kubernetes.io/projected/8535db98-63f7-42ff-9774-4d864939faf7-kube-api-access-7dr8v\") pod \"kuadrant-operator-controller-manager-86878c769f-pw46f\" (UID: \"8535db98-63f7-42ff-9774-4d864939faf7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.537701 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.537603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:51.684750 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:51.684724 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f"] Jul 15 14:53:51.687207 ip-10-0-141-12 kubenswrapper[2567]: W0715 14:53:51.687181 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535db98_63f7_42ff_9774_4d864939faf7.slice/crio-a48a6bc5ef80253b85539fd3e17dadbbddfcedf135862a0cb123205c3599dde6 WatchSource:0}: Error finding container a48a6bc5ef80253b85539fd3e17dadbbddfcedf135862a0cb123205c3599dde6: Status 404 returned error can't find the container with id a48a6bc5ef80253b85539fd3e17dadbbddfcedf135862a0cb123205c3599dde6 Jul 15 14:53:52.141657 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:52.141622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" event={"ID":"8535db98-63f7-42ff-9774-4d864939faf7","Type":"ContainerStarted","Data":"d3e5babc1784118b666de12d15909437a66dbd4e952860bf82734f648f65ce41"} Jul 15 14:53:52.141855 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:52.141672 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" event={"ID":"8535db98-63f7-42ff-9774-4d864939faf7","Type":"ContainerStarted","Data":"a48a6bc5ef80253b85539fd3e17dadbbddfcedf135862a0cb123205c3599dde6"} Jul 15 14:53:52.141855 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:52.141712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:53:52.160810 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:53:52.160743 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" podStartSLOduration=1.160722489 podStartE2EDuration="1.160722489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:53:52.160164694 +0000 UTC m=+1202.569783973" watchObservedRunningTime="2026-07-15 14:53:52.160722489 +0000 UTC m=+1202.570341770" Jul 15 14:54:03.149863 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:54:03.149831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pw46f" Jul 15 14:58:50.126794 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:58:50.126765 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 14:58:50.133223 ip-10-0-141-12 kubenswrapper[2567]: I0715 14:58:50.133204 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:00:00.143366 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.143329 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm"] Jul 15 15:00:00.147203 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.147180 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" Jul 15 15:00:00.149243 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.149223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:00:00.156433 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.156416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" Jul 15 15:00:00.165896 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.165869 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm"] Jul 15 15:00:00.295289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.295262 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm"] Jul 15 15:00:00.296268 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:00:00.296239 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678de9be_6c9b_456d_8109_ea1042a5503f.slice/crio-ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8 WatchSource:0}: Error finding container ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8: Status 404 returned error can't find the container with id ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8 Jul 15 15:00:00.298076 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:00.298039 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 15:00:01.080629 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:01.080597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" event={"ID":"678de9be-6c9b-456d-8109-ea1042a5503f","Type":"ContainerStarted","Data":"ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8"} Jul 15 15:00:14.788475 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.788442 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:00:14.793669 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.793645 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.795577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.795557 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8a5fa064-serving-cert\"" Jul 15 15:00:14.801712 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.801688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:00:14.837294 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.837262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.837472 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.837324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.837472 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.837361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q5g\" (UniqueName: \"kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.937781 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.937742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.937981 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.937803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.937981 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.937850 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q5g\" (UniqueName: \"kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.937981 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:00:14.937893 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-serving-cert: secret "maas-api-e2e-ait-8a5fa064-serving-cert" not found Jul 15 15:00:14.937981 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:00:14.937958 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls podName:b7d4c9d2-56ca-4733-93da-f07ee3a23a94 nodeName:}" failed. No retries permitted until 2026-07-15 15:00:15.437935991 +0000 UTC m=+1585.847555263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls") pod "maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" (UID: "b7d4c9d2-56ca-4733-93da-f07ee3a23a94") : secret "maas-api-e2e-ait-8a5fa064-serving-cert" not found Jul 15 15:00:14.938606 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.938585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:14.946451 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:14.946421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q5g\" (UniqueName: \"kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:15.442355 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:15.442316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:15.444763 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:15.444732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") pod \"maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:15.706200 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:15.706100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:15.855672 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:15.855649 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:00:15.857914 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:00:15.857884 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d4c9d2_56ca_4733_93da_f07ee3a23a94.slice/crio-d5878ae82d5ead28ea792eeafb245a3931aa9a374194a2dbd7c5ff4517cf9e43 WatchSource:0}: Error finding container d5878ae82d5ead28ea792eeafb245a3931aa9a374194a2dbd7c5ff4517cf9e43: Status 404 returned error can't find the container with id d5878ae82d5ead28ea792eeafb245a3931aa9a374194a2dbd7c5ff4517cf9e43 Jul 15 15:00:16.157757 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:16.157722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" event={"ID":"b7d4c9d2-56ca-4733-93da-f07ee3a23a94","Type":"ContainerStarted","Data":"d5878ae82d5ead28ea792eeafb245a3931aa9a374194a2dbd7c5ff4517cf9e43"} Jul 15 15:00:17.164098 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:17.164044 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" event={"ID":"b7d4c9d2-56ca-4733-93da-f07ee3a23a94","Type":"ContainerStarted","Data":"1c82b88c1cce876ad66e5b6d645e055fbb3ebb5c040475e985cbb8259caaf0c3"} Jul 15 15:00:17.164521 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:17.164280 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:17.182577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:17.182511 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" podStartSLOduration=2.633437292 podStartE2EDuration="3.182494059s" totalImagesPullingTime="549.056767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:00:17.181418025 +0000 UTC m=+1587.591037302" watchObservedRunningTime="2026-07-15 15:00:17.182494059 +0000 UTC m=+1587.592113352" Jul 15 15:00:23.175080 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:23.174983 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:00:23.198394 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:23.198350 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" event={"ID":"678de9be-6c9b-456d-8109-ea1042a5503f","Type":"ContainerStarted","Data":"fd554bbd19e110872f652121d11a5149ee8179f53c9420ab529b482ece49c615"} Jul 15 15:00:23.213896 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:23.213831 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" podStartSLOduration=0.967252093 podStartE2EDuration="23.213792249s" totalImagesPullingTime="22.246540156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:00:23.212110672 +0000 UTC m=+1593.621729951" watchObservedRunningTime="2026-07-15 15:00:23.213792249 +0000 UTC m=+1593.623411528" Jul 15 15:00:24.205360 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:24.205323 2567 generic.go:356] "Generic (PLEG): container finished" podID="678de9be-6c9b-456d-8109-ea1042a5503f" containerID="fd554bbd19e110872f652121d11a5149ee8179f53c9420ab529b482ece49c615" exitCode=0 Jul 15 15:00:24.205729 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:24.205371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" event={"ID":"678de9be-6c9b-456d-8109-ea1042a5503f","Type":"ContainerDied","Data":"fd554bbd19e110872f652121d11a5149ee8179f53c9420ab529b482ece49c615"} Jul 15 15:00:25.351806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:25.351779 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" Jul 15 15:00:26.218344 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:26.218309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" event={"ID":"678de9be-6c9b-456d-8109-ea1042a5503f","Type":"ContainerDied","Data":"ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8"} Jul 15 15:00:26.218528 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:26.218350 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735460-pmlbm" Jul 15 15:00:26.218528 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:00:26.218362 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee81fe56cbd1873e327387132c2daf953afdaef24cab01b6d690abbf527e12f8" Jul 15 15:01:19.949504 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.949472 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc"] Jul 15 15:01:19.949934 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.949875 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678de9be-6c9b-456d-8109-ea1042a5503f" containerName="cleanup" Jul 15 15:01:19.949934 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.949887 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="678de9be-6c9b-456d-8109-ea1042a5503f" containerName="cleanup" Jul 15 15:01:19.950005 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.949954 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="678de9be-6c9b-456d-8109-ea1042a5503f" containerName="cleanup" Jul 15 15:01:19.957993 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.957326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:19.959528 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.959448 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:01:19.960488 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:19.960421 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc"] Jul 15 15:01:20.037283 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.037262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc\" (UID: \"d33a69da-8f87-425b-b2ea-c665d3377be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:20.138514 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.138487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc\" (UID: \"d33a69da-8f87-425b-b2ea-c665d3377be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:20.139077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.139039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc\" (UID: \"d33a69da-8f87-425b-b2ea-c665d3377be8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:20.269878 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.269828 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:20.402947 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.402916 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc"] Jul 15 15:01:20.403086 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:01:20.402992 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33a69da_8f87_425b_b2ea_c665d3377be8.slice/crio-4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b WatchSource:0}: Error finding container 4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b: Status 404 returned error can't find the container with id 4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b Jul 15 15:01:20.498662 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:20.498640 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" event={"ID":"d33a69da-8f87-425b-b2ea-c665d3377be8","Type":"ContainerStarted","Data":"4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b"} Jul 15 15:01:21.507151 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:21.507116 2567 generic.go:356] "Generic (PLEG): container finished" podID="d33a69da-8f87-425b-b2ea-c665d3377be8" containerID="563d6b589e7ced7cb46547e0aa4c35a2bee61a451186be50ed3c2d71a6414197" exitCode=0 Jul 15 15:01:21.507513 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:21.507203 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" event={"ID":"d33a69da-8f87-425b-b2ea-c665d3377be8","Type":"ContainerDied","Data":"563d6b589e7ced7cb46547e0aa4c35a2bee61a451186be50ed3c2d71a6414197"} Jul 15 15:01:22.639635 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:22.639609 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:22.760740 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:22.760710 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca\") pod \"d33a69da-8f87-425b-b2ea-c665d3377be8\" (UID: \"d33a69da-8f87-425b-b2ea-c665d3377be8\") " Jul 15 15:01:22.761033 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:22.761011 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d33a69da-8f87-425b-b2ea-c665d3377be8" (UID: "d33a69da-8f87-425b-b2ea-c665d3377be8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:01:22.861702 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:22.861675 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a69da-8f87-425b-b2ea-c665d3377be8-maas-api-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:01:23.525943 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:23.525911 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" Jul 15 15:01:23.526140 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:23.525946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc" event={"ID":"d33a69da-8f87-425b-b2ea-c665d3377be8","Type":"ContainerDied","Data":"4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b"} Jul 15 15:01:23.526140 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:23.526018 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cccbc6ef83dbbd96a83a4814ee489c861b471b783496dfa11c1e4308840320b" Jul 15 15:01:30.404165 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.404125 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:01:30.404732 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.404439 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" podUID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" containerName="maas-api" containerID="cri-o://1c82b88c1cce876ad66e5b6d645e055fbb3ebb5c040475e985cbb8259caaf0c3" gracePeriod=30 Jul 15 15:01:30.566481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.566438 2567 generic.go:356] "Generic (PLEG): container finished" podID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" containerID="1c82b88c1cce876ad66e5b6d645e055fbb3ebb5c040475e985cbb8259caaf0c3" exitCode=0 Jul 15 15:01:30.566628 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.566489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" event={"ID":"b7d4c9d2-56ca-4733-93da-f07ee3a23a94","Type":"ContainerDied","Data":"1c82b88c1cce876ad66e5b6d645e055fbb3ebb5c040475e985cbb8259caaf0c3"} Jul 15 15:01:30.664401 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.664342 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:01:30.825394 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.825365 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42q5g\" (UniqueName: \"kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g\") pod \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " Jul 15 15:01:30.825548 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.825466 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") pod \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " Jul 15 15:01:30.825548 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.825492 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca\") pod \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\" (UID: \"b7d4c9d2-56ca-4733-93da-f07ee3a23a94\") " Jul 15 15:01:30.825892 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.825864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7d4c9d2-56ca-4733-93da-f07ee3a23a94" (UID: "b7d4c9d2-56ca-4733-93da-f07ee3a23a94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:01:30.827506 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.827485 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7d4c9d2-56ca-4733-93da-f07ee3a23a94" (UID: "b7d4c9d2-56ca-4733-93da-f07ee3a23a94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:01:30.827605 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.827588 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g" (OuterVolumeSpecName: "kube-api-access-42q5g") pod "b7d4c9d2-56ca-4733-93da-f07ee3a23a94" (UID: "b7d4c9d2-56ca-4733-93da-f07ee3a23a94"). InnerVolumeSpecName "kube-api-access-42q5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:01:30.926107 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.926030 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:01:30.926107 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.926072 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:01:30.926107 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:30.926082 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42q5g\" (UniqueName: \"kubernetes.io/projected/b7d4c9d2-56ca-4733-93da-f07ee3a23a94-kube-api-access-42q5g\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:01:31.573389 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:31.573365 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" Jul 15 15:01:31.573782 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:31.573403 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b" event={"ID":"b7d4c9d2-56ca-4733-93da-f07ee3a23a94","Type":"ContainerDied","Data":"d5878ae82d5ead28ea792eeafb245a3931aa9a374194a2dbd7c5ff4517cf9e43"} Jul 15 15:01:31.573782 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:31.573462 2567 scope.go:117] "RemoveContainer" containerID="1c82b88c1cce876ad66e5b6d645e055fbb3ebb5c040475e985cbb8259caaf0c3" Jul 15 15:01:31.596072 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:31.596033 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:01:31.600728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:31.600705 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a5fa064-76c7695bf8-s229b"] Jul 15 15:01:32.165024 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:32.164992 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" path="/var/lib/kubelet/pods/b7d4c9d2-56ca-4733-93da-f07ee3a23a94/volumes" Jul 15 15:01:47.214376 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214342 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:01:47.214847 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214833 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33a69da-8f87-425b-b2ea-c665d3377be8" containerName="revoke-keys" Jul 15 15:01:47.214903 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214850 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33a69da-8f87-425b-b2ea-c665d3377be8" containerName="revoke-keys" Jul 15 15:01:47.214903 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214889 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" containerName="maas-api" Jul 15 15:01:47.214903 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214897 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" containerName="maas-api" Jul 15 15:01:47.215012 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214966 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d4c9d2-56ca-4733-93da-f07ee3a23a94" containerName="maas-api" Jul 15 15:01:47.215012 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.214981 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d33a69da-8f87-425b-b2ea-c665d3377be8" containerName="revoke-keys" Jul 15 15:01:47.219597 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.219576 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.221752 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.221728 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fc2c3cfa-serving-cert\"" Jul 15 15:01:47.231096 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.231075 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:01:47.259936 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.259913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8rx\" (UniqueName: \"kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.260091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.259952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.260091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.259988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.360604 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.360580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8rx\" (UniqueName: \"kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.360738 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.360622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.360738 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.360648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.361387 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.361364 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.363094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.363075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.369186 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.369163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8rx\" (UniqueName: \"kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx\") pod \"maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.534865 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.534806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:47.881542 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:47.881516 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:01:47.883213 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:01:47.883183 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5601da13_bbec_42bb_9bf0_bb7a0d003bb6.slice/crio-95736595767e736589f08d4a8c45bc8b09a778ccdba1e09e5525f4efe1e7f21b WatchSource:0}: Error finding container 95736595767e736589f08d4a8c45bc8b09a778ccdba1e09e5525f4efe1e7f21b: Status 404 returned error can't find the container with id 95736595767e736589f08d4a8c45bc8b09a778ccdba1e09e5525f4efe1e7f21b Jul 15 15:01:48.662246 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:48.662161 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" event={"ID":"5601da13-bbec-42bb-9bf0-bb7a0d003bb6","Type":"ContainerStarted","Data":"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f"} Jul 15 15:01:48.662246 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:48.662211 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" event={"ID":"5601da13-bbec-42bb-9bf0-bb7a0d003bb6","Type":"ContainerStarted","Data":"95736595767e736589f08d4a8c45bc8b09a778ccdba1e09e5525f4efe1e7f21b"} Jul 15 15:01:48.662745 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:48.662265 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:01:48.680935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:48.680890 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" podStartSLOduration=1.22110577 podStartE2EDuration="1.680875082s" totalImagesPullingTime="459.769312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:01:48.678095026 +0000 UTC m=+1679.087714304" watchObservedRunningTime="2026-07-15 15:01:48.680875082 +0000 UTC m=+1679.090494365" Jul 15 15:01:54.674781 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:01:54.674746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:02:41.719277 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.719234 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd"] Jul 15 15:02:41.722791 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.722765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:41.724849 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.724824 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:02:41.730800 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.730773 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd"] Jul 15 15:02:41.827432 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.827394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd\" (UID: \"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:41.928962 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.928919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd\" (UID: \"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:41.929651 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:41.929628 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd\" (UID: \"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:42.036237 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:42.036144 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:42.180620 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:42.180595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd"] Jul 15 15:02:42.182451 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:02:42.182424 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc0ccfc_98f0_4e25_9abb_5011b7492fe3.slice/crio-fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54 WatchSource:0}: Error finding container fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54: Status 404 returned error can't find the container with id fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54 Jul 15 15:02:42.952409 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:42.952372 2567 generic.go:356] "Generic (PLEG): container finished" podID="ecc0ccfc-98f0-4e25-9abb-5011b7492fe3" containerID="ef3e645fb1816c28700d152f483440fbeb6c9c8b8611cca1bb7fbc7bccf917ae" exitCode=0 Jul 15 15:02:42.952850 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:42.952425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" event={"ID":"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3","Type":"ContainerDied","Data":"ef3e645fb1816c28700d152f483440fbeb6c9c8b8611cca1bb7fbc7bccf917ae"} Jul 15 15:02:42.952850 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:42.952491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" event={"ID":"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3","Type":"ContainerStarted","Data":"fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54"} Jul 15 15:02:44.089077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.089022 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:44.252844 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.252748 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca\") pod \"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3\" (UID: \"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3\") " Jul 15 15:02:44.253197 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.253176 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ecc0ccfc-98f0-4e25-9abb-5011b7492fe3" (UID: "ecc0ccfc-98f0-4e25-9abb-5011b7492fe3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:02:44.354444 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.354405 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0ccfc-98f0-4e25-9abb-5011b7492fe3-maas-api-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:02:44.964496 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.964468 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" Jul 15 15:02:44.964666 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.964513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd" event={"ID":"ecc0ccfc-98f0-4e25-9abb-5011b7492fe3","Type":"ContainerDied","Data":"fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54"} Jul 15 15:02:44.964666 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:44.964557 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe79f75cd3caa065914d523dd0466001d6880e2fdb51141c265b75caa39b4d54" Jul 15 15:02:52.294705 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.294664 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:02:52.295176 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.294927 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" podUID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" containerName="maas-api" containerID="cri-o://087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f" gracePeriod=30 Jul 15 15:02:52.557902 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.557879 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:02:52.630129 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.630093 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls\") pod \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " Jul 15 15:02:52.630129 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.630151 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca\") pod \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " Jul 15 15:02:52.630387 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.630197 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w8rx\" (UniqueName: \"kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx\") pod \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\" (UID: \"5601da13-bbec-42bb-9bf0-bb7a0d003bb6\") " Jul 15 15:02:52.630592 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.630564 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5601da13-bbec-42bb-9bf0-bb7a0d003bb6" (UID: "5601da13-bbec-42bb-9bf0-bb7a0d003bb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:02:52.632281 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.632247 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5601da13-bbec-42bb-9bf0-bb7a0d003bb6" (UID: "5601da13-bbec-42bb-9bf0-bb7a0d003bb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:02:52.632393 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.632352 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx" (OuterVolumeSpecName: "kube-api-access-7w8rx") pod "5601da13-bbec-42bb-9bf0-bb7a0d003bb6" (UID: "5601da13-bbec-42bb-9bf0-bb7a0d003bb6"). InnerVolumeSpecName "kube-api-access-7w8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:02:52.731020 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.730982 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:02:52.731020 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.731020 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:02:52.731234 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:52.731035 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w8rx\" (UniqueName: \"kubernetes.io/projected/5601da13-bbec-42bb-9bf0-bb7a0d003bb6-kube-api-access-7w8rx\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:02:53.010309 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.010222 2567 generic.go:356] "Generic (PLEG): container finished" podID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" containerID="087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f" exitCode=0 Jul 15 15:02:53.010309 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.010267 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" Jul 15 15:02:53.010495 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.010310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" event={"ID":"5601da13-bbec-42bb-9bf0-bb7a0d003bb6","Type":"ContainerDied","Data":"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f"} Jul 15 15:02:53.010495 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.010359 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z" event={"ID":"5601da13-bbec-42bb-9bf0-bb7a0d003bb6","Type":"ContainerDied","Data":"95736595767e736589f08d4a8c45bc8b09a778ccdba1e09e5525f4efe1e7f21b"} Jul 15 15:02:53.010495 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.010381 2567 scope.go:117] "RemoveContainer" containerID="087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f" Jul 15 15:02:53.019326 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.019310 2567 scope.go:117] "RemoveContainer" containerID="087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f" Jul 15 15:02:53.019552 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:02:53.019534 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f\": container with ID starting with 087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f not found: ID does not exist" containerID="087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f" Jul 15 15:02:53.019594 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.019561 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f"} err="failed to get container status \"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f\": rpc error: code = NotFound desc = could not find container \"087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f\": container with ID starting with 087902f16916a4608e34f9606d4d1415efd794cedef2891f6272c6560899695f not found: ID does not exist" Jul 15 15:02:53.032439 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.032419 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:02:53.037313 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:53.037289 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc2c3cfa-747547d779-lnw8z"] Jul 15 15:02:54.163895 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:54.163861 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" path="/var/lib/kubelet/pods/5601da13-bbec-42bb-9bf0-bb7a0d003bb6/volumes" Jul 15 15:02:59.802370 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.802335 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:02:59.802909 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.802888 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" containerName="maas-api" Jul 15 15:02:59.802978 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.802913 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" containerName="maas-api" Jul 15 15:02:59.802978 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.802947 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc0ccfc-98f0-4e25-9abb-5011b7492fe3" containerName="revoke-keys" Jul 15 15:02:59.802978 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.802955 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc0ccfc-98f0-4e25-9abb-5011b7492fe3" containerName="revoke-keys" Jul 15 15:02:59.803162 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.803039 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc0ccfc-98f0-4e25-9abb-5011b7492fe3" containerName="revoke-keys" Jul 15 15:02:59.803162 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.803052 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5601da13-bbec-42bb-9bf0-bb7a0d003bb6" containerName="maas-api" Jul 15 15:02:59.807829 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.807808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.809745 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.809719 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a418840f-serving-cert\"" Jul 15 15:02:59.815324 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.815267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:02:59.892549 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.892526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.892704 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.892568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6ql\" (UniqueName: \"kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.892704 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.892593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.993459 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.993428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.993602 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.993471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6ql\" (UniqueName: \"kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.993602 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.993493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:02:59.993602 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:02:59.993582 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-serving-cert: secret "maas-api-e2e-ait-a418840f-serving-cert" not found Jul 15 15:02:59.993767 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:02:59.993629 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls podName:a13595ce-7589-41d9-a12d-8404741c6557 nodeName:}" failed. No retries permitted until 2026-07-15 15:03:00.493615776 +0000 UTC m=+1750.903235033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls") pod "maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" (UID: "a13595ce-7589-41d9-a12d-8404741c6557") : secret "maas-api-e2e-ait-a418840f-serving-cert" not found Jul 15 15:02:59.994024 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:02:59.994004 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:00.002117 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:00.002099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6ql\" (UniqueName: \"kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:00.497206 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:00.497177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:00.499490 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:00.499469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") pod \"maas-api-e2e-ait-a418840f-577679bcc9-w5lmp\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:00.721692 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:00.721657 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:00.862317 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:00.862295 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:03:00.863866 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:03:00.863840 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13595ce_7589_41d9_a12d_8404741c6557.slice/crio-095dbd1789f10dcf871c4aad39c908325f66413f355961ff48f5d498f67ce121 WatchSource:0}: Error finding container 095dbd1789f10dcf871c4aad39c908325f66413f355961ff48f5d498f67ce121: Status 404 returned error can't find the container with id 095dbd1789f10dcf871c4aad39c908325f66413f355961ff48f5d498f67ce121 Jul 15 15:03:01.052511 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:01.052483 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" event={"ID":"a13595ce-7589-41d9-a12d-8404741c6557","Type":"ContainerStarted","Data":"095dbd1789f10dcf871c4aad39c908325f66413f355961ff48f5d498f67ce121"} Jul 15 15:03:02.059930 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:02.059888 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" event={"ID":"a13595ce-7589-41d9-a12d-8404741c6557","Type":"ContainerStarted","Data":"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90"} Jul 15 15:03:02.060471 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:02.060094 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:02.078978 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:02.078928 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" podStartSLOduration=2.6172310530000003 podStartE2EDuration="3.07891341s" totalImagesPullingTime="461.682357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:03:02.075592358 +0000 UTC m=+1752.485211638" watchObservedRunningTime="2026-07-15 15:03:02.07891341 +0000 UTC m=+1752.488532687" Jul 15 15:03:04.838774 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:04.838735 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:04.842310 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:04.842287 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:04.846212 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:04.846190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:03:04.849839 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:04.849818 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:04.932643 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:04.932619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a418840f-5nqgh\" (UID: \"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:05.033763 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:05.033735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a418840f-5nqgh\" (UID: \"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:05.034351 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:05.034333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a418840f-5nqgh\" (UID: \"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:05.157393 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:05.157331 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:05.292827 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:05.292800 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:05.294725 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:03:05.294701 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd31c8e_c5cd_44d0_989b_3fbc1fe15162.slice/crio-c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a WatchSource:0}: Error finding container c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a: Status 404 returned error can't find the container with id c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a Jul 15 15:03:06.081749 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:06.081714 2567 generic.go:356] "Generic (PLEG): container finished" podID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerID="a3f44f2f0ebb671418108016bd6e337a0d9a0f4ff30af9a14b32de61252f1f23" exitCode=7 Jul 15 15:03:06.082085 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:06.081795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" event={"ID":"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162","Type":"ContainerDied","Data":"a3f44f2f0ebb671418108016bd6e337a0d9a0f4ff30af9a14b32de61252f1f23"} Jul 15 15:03:06.082085 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:06.081855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" event={"ID":"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162","Type":"ContainerStarted","Data":"c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a"} Jul 15 15:03:06.082085 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:06.081957 2567 scope.go:117] "RemoveContainer" containerID="a3f44f2f0ebb671418108016bd6e337a0d9a0f4ff30af9a14b32de61252f1f23" Jul 15 15:03:07.094839 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:07.094807 2567 generic.go:356] "Generic (PLEG): container finished" podID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerID="045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e" exitCode=7 Jul 15 15:03:07.095292 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:07.094900 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" event={"ID":"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162","Type":"ContainerDied","Data":"045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e"} Jul 15 15:03:07.095292 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:07.094996 2567 scope.go:117] "RemoveContainer" containerID="a3f44f2f0ebb671418108016bd6e337a0d9a0f4ff30af9a14b32de61252f1f23" Jul 15 15:03:07.095292 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:07.095042 2567 scope.go:117] "RemoveContainer" containerID="045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e" Jul 15 15:03:07.095292 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:03:07.095254 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a418840f-5nqgh_odh-ai-gateway-infra(3fd31c8e-c5cd-44d0-989b-3fbc1fe15162)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" Jul 15 15:03:08.070413 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:08.070387 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:03:08.104507 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:08.104483 2567 scope.go:117] "RemoveContainer" containerID="045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e" Jul 15 15:03:08.104944 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:03:08.104685 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a418840f-5nqgh_odh-ai-gateway-infra(3fd31c8e-c5cd-44d0-989b-3fbc1fe15162)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" Jul 15 15:03:21.159341 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:21.159305 2567 scope.go:117] "RemoveContainer" containerID="045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e" Jul 15 15:03:22.181512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:22.181434 2567 generic.go:356] "Generic (PLEG): container finished" podID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerID="03688b4ce09138fc1054f14ca5806518966944250c56dd3409d94d55f50592e5" exitCode=0 Jul 15 15:03:22.181512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:22.181497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" event={"ID":"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162","Type":"ContainerDied","Data":"03688b4ce09138fc1054f14ca5806518966944250c56dd3409d94d55f50592e5"} Jul 15 15:03:22.181923 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:22.181543 2567 scope.go:117] "RemoveContainer" containerID="045a490b25afee4ccf155e7a4fe97d936e2ffc72f8efa30117547b17cc91c73e" Jul 15 15:03:23.217123 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:23.217089 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:23.327900 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:23.327871 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:23.483664 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:23.483564 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca\") pod \"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162\" (UID: \"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162\") " Jul 15 15:03:23.483921 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:23.483900 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" (UID: "3fd31c8e-c5cd-44d0-989b-3fbc1fe15162"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:03:23.584492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:23.584447 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162-maas-api-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:03:24.195598 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:24.195562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" event={"ID":"3fd31c8e-c5cd-44d0-989b-3fbc1fe15162","Type":"ContainerDied","Data":"c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a"} Jul 15 15:03:24.195598 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:24.195589 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh" Jul 15 15:03:24.195824 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:24.195612 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e2d4e8948e1c13f5b1c8a44acbfe50e3d710f6d789dcb324f09a174175090a" Jul 15 15:03:24.228732 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:24.228699 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:24.234004 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:24.233971 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a418840f-5nqgh"] Jul 15 15:03:26.164378 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:26.164341 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" path="/var/lib/kubelet/pods/3fd31c8e-c5cd-44d0-989b-3fbc1fe15162/volumes" Jul 15 15:03:50.162569 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:50.162472 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:03:50.170037 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:03:50.170023 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:04:05.527672 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.527637 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:04:05.528172 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.527877 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" podUID="a13595ce-7589-41d9-a12d-8404741c6557" containerName="maas-api" containerID="cri-o://da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90" gracePeriod=30 Jul 15 15:04:05.775876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.775848 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:04:05.825965 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.825918 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn6ql\" (UniqueName: \"kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql\") pod \"a13595ce-7589-41d9-a12d-8404741c6557\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " Jul 15 15:04:05.826207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.826087 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") pod \"a13595ce-7589-41d9-a12d-8404741c6557\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " Jul 15 15:04:05.826207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.826157 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca\") pod \"a13595ce-7589-41d9-a12d-8404741c6557\" (UID: \"a13595ce-7589-41d9-a12d-8404741c6557\") " Jul 15 15:04:05.826521 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.826485 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a13595ce-7589-41d9-a12d-8404741c6557" (UID: "a13595ce-7589-41d9-a12d-8404741c6557"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:04:05.828194 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.828164 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql" (OuterVolumeSpecName: "kube-api-access-mn6ql") pod "a13595ce-7589-41d9-a12d-8404741c6557" (UID: "a13595ce-7589-41d9-a12d-8404741c6557"). InnerVolumeSpecName "kube-api-access-mn6ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:04:05.828315 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.828263 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a13595ce-7589-41d9-a12d-8404741c6557" (UID: "a13595ce-7589-41d9-a12d-8404741c6557"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:04:05.927203 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.927162 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13595ce-7589-41d9-a12d-8404741c6557-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:04:05.927203 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.927196 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13595ce-7589-41d9-a12d-8404741c6557-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:04:05.927203 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:05.927206 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6ql\" (UniqueName: \"kubernetes.io/projected/a13595ce-7589-41d9-a12d-8404741c6557-kube-api-access-mn6ql\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:04:06.428391 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.428292 2567 generic.go:356] "Generic (PLEG): container finished" podID="a13595ce-7589-41d9-a12d-8404741c6557" containerID="da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90" exitCode=0 Jul 15 15:04:06.428391 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.428341 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" Jul 15 15:04:06.428601 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.428383 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" event={"ID":"a13595ce-7589-41d9-a12d-8404741c6557","Type":"ContainerDied","Data":"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90"} Jul 15 15:04:06.428601 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.428442 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp" event={"ID":"a13595ce-7589-41d9-a12d-8404741c6557","Type":"ContainerDied","Data":"095dbd1789f10dcf871c4aad39c908325f66413f355961ff48f5d498f67ce121"} Jul 15 15:04:06.428601 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.428466 2567 scope.go:117] "RemoveContainer" containerID="da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90" Jul 15 15:04:06.438137 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.438111 2567 scope.go:117] "RemoveContainer" containerID="da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90" Jul 15 15:04:06.438443 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:04:06.438422 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90\": container with ID starting with da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90 not found: ID does not exist" containerID="da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90" Jul 15 15:04:06.438516 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.438457 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90"} err="failed to get container status \"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90\": rpc error: code = NotFound desc = could not find container \"da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90\": container with ID starting with da0b1152b13d99db66a2c6d14e886e680a780fb68baaaf6216ff7f03da2d0a90 not found: ID does not exist" Jul 15 15:04:06.447884 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.447853 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:04:06.450324 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:06.450301 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a418840f-577679bcc9-w5lmp"] Jul 15 15:04:08.164476 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:08.164443 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13595ce-7589-41d9-a12d-8404741c6557" path="/var/lib/kubelet/pods/a13595ce-7589-41d9-a12d-8404741c6557/volumes" Jul 15 15:04:21.338334 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338280 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:04:21.338917 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338841 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.338917 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338861 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.338917 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338884 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13595ce-7589-41d9-a12d-8404741c6557" containerName="maas-api" Jul 15 15:04:21.338917 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338894 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13595ce-7589-41d9-a12d-8404741c6557" containerName="maas-api" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338921 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.338929 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.339006 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.339022 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.339030 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:21.339179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.339039 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13595ce-7589-41d9-a12d-8404741c6557" containerName="maas-api" Jul 15 15:04:21.343557 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.343533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.345531 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.345510 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9c3abd0e-openshift-default-dockercfg-kdwsr\"" Jul 15 15:04:21.352769 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.352270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:04:21.477384 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnzr\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.477728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.477634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578286 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578467 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578514 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578628 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnzr\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578647 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578788 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578788 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578895 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578847 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.578895 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.578626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.579038 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.579016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.579134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.579095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.579321 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.579300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.580831 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.580808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.581126 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.581110 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.590624 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.590568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.590711 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.590659 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnzr\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr\") pod \"e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.657352 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.657316 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:21.802189 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:21.802162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:04:21.804130 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:04:21.804103 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092a9570_9b25_4a19_aed4_2a23ee6e45ca.slice/crio-92c0825e84563b293f8b2a3cdb25af68f7549e87a5db52e3eced98884a528ffb WatchSource:0}: Error finding container 92c0825e84563b293f8b2a3cdb25af68f7549e87a5db52e3eced98884a528ffb: Status 404 returned error can't find the container with id 92c0825e84563b293f8b2a3cdb25af68f7549e87a5db52e3eced98884a528ffb Jul 15 15:04:22.523668 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.523632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerStarted","Data":"92c0825e84563b293f8b2a3cdb25af68f7549e87a5db52e3eced98884a528ffb"} Jul 15 15:04:22.710210 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.710174 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:04:22.711479 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.711456 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:22.711657 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.711627 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd31c8e-c5cd-44d0-989b-3fbc1fe15162" containerName="revoke-keys" Jul 15 15:04:22.718404 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.718377 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.721126 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.720883 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9c3abd0e-serving-cert\"" Jul 15 15:04:22.734133 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.733614 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:04:22.889502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.889429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.889502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.889483 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8tb2\" (UniqueName: \"kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.889769 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.889517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.990458 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.990412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.990458 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.990465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8tb2\" (UniqueName: \"kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.990725 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.990493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.991471 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.991441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:22.993104 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:22.993084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:23.000200 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:23.000134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8tb2\" (UniqueName: \"kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2\") pod \"maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:23.041259 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:23.041220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:23.206164 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:23.206118 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:04:23.207469 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:04:23.207438 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d810a05_203e_4b5e_8733_2cc1de1cb698.slice/crio-5f01a03fb467e947b13db007af0f55c260a80aba853c62698fb7a7a9c8fb3c90 WatchSource:0}: Error finding container 5f01a03fb467e947b13db007af0f55c260a80aba853c62698fb7a7a9c8fb3c90: Status 404 returned error can't find the container with id 5f01a03fb467e947b13db007af0f55c260a80aba853c62698fb7a7a9c8fb3c90 Jul 15 15:04:23.530879 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:23.530772 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" event={"ID":"5d810a05-203e-4b5e-8733-2cc1de1cb698","Type":"ContainerStarted","Data":"5f01a03fb467e947b13db007af0f55c260a80aba853c62698fb7a7a9c8fb3c90"} Jul 15 15:04:24.614465 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:24.614427 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:24.614851 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:24.614510 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:24.614851 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:24.614540 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:25.543895 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.543861 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" event={"ID":"5d810a05-203e-4b5e-8733-2cc1de1cb698","Type":"ContainerStarted","Data":"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5"} Jul 15 15:04:25.544135 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.543936 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:25.545557 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.545536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerStarted","Data":"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252"} Jul 15 15:04:25.560490 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.560389 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" podStartSLOduration=2.155794459 podStartE2EDuration="3.560372307s" totalImagesPullingTime="1.404577848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:04:25.559614133 +0000 UTC m=+1835.969233413" watchObservedRunningTime="2026-07-15 15:04:25.560372307 +0000 UTC m=+1835.969991589" Jul 15 15:04:25.578195 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.578149 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podStartSLOduration=1.770009558 podStartE2EDuration="4.578133039s" totalImagesPullingTime="2.808123481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:04:25.576340563 +0000 UTC m=+1835.985959841" watchObservedRunningTime="2026-07-15 15:04:25.578133039 +0000 UTC m=+1835.987752386" Jul 15 15:04:25.657800 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.657762 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:25.662817 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:25.662793 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:26.551445 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:26.551410 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:26.552554 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:26.552533 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:31.558086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:31.558041 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:04:34.604484 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.604452 2567 generic.go:356] "Generic (PLEG): container finished" podID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerID="3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252" exitCode=0 Jul 15 15:04:34.604910 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.604497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerDied","Data":"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252"} Jul 15 15:04:34.604910 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.604812 2567 scope.go:117] "RemoveContainer" containerID="3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252" Jul 15 15:04:34.606708 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.606682 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:34.606785 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.606757 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:34.606828 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:34.606811 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:04:35.612041 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:35.611987 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerStarted","Data":"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45"} Jul 15 15:04:35.658475 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:35.658433 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:35.658754 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:35.658731 2567 patch_prober.go:28] interesting pod/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 15:04:35.658825 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:35.658802 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 15:04:36.658795 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:36.658752 2567 patch_prober.go:28] interesting pod/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 15:04:36.659328 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:36.658848 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 15:04:37.658698 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:37.658664 2567 patch_prober.go:28] interesting pod/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 15:04:37.658897 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:37.658724 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 15:04:38.658538 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:38.658501 2567 patch_prober.go:28] interesting pod/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 15:04:38.658713 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:38.658562 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 15:04:39.662923 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:39.662892 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:39.663322 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:39.663171 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:04:39.663937 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:04:39.663917 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:05:37.519037 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.519002 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:05:37.519563 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.519463 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" podUID="5d810a05-203e-4b5e-8733-2cc1de1cb698" containerName="maas-api" containerID="cri-o://f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5" gracePeriod=30 Jul 15 15:05:37.766661 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.766637 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:05:37.909598 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.909568 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca\") pod \"5d810a05-203e-4b5e-8733-2cc1de1cb698\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " Jul 15 15:05:37.909803 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.909634 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls\") pod \"5d810a05-203e-4b5e-8733-2cc1de1cb698\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " Jul 15 15:05:37.909803 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.909651 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8tb2\" (UniqueName: \"kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2\") pod \"5d810a05-203e-4b5e-8733-2cc1de1cb698\" (UID: \"5d810a05-203e-4b5e-8733-2cc1de1cb698\") " Jul 15 15:05:37.910022 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.909997 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d810a05-203e-4b5e-8733-2cc1de1cb698" (UID: "5d810a05-203e-4b5e-8733-2cc1de1cb698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:05:37.911663 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.911645 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d810a05-203e-4b5e-8733-2cc1de1cb698" (UID: "5d810a05-203e-4b5e-8733-2cc1de1cb698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:05:37.911758 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.911745 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2" (OuterVolumeSpecName: "kube-api-access-m8tb2") pod "5d810a05-203e-4b5e-8733-2cc1de1cb698" (UID: "5d810a05-203e-4b5e-8733-2cc1de1cb698"). InnerVolumeSpecName "kube-api-access-m8tb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:05:37.952747 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.952716 2567 generic.go:356] "Generic (PLEG): container finished" podID="5d810a05-203e-4b5e-8733-2cc1de1cb698" containerID="f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5" exitCode=0 Jul 15 15:05:37.952876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.952759 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" Jul 15 15:05:37.952876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.952804 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" event={"ID":"5d810a05-203e-4b5e-8733-2cc1de1cb698","Type":"ContainerDied","Data":"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5"} Jul 15 15:05:37.952876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.952854 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf" event={"ID":"5d810a05-203e-4b5e-8733-2cc1de1cb698","Type":"ContainerDied","Data":"5f01a03fb467e947b13db007af0f55c260a80aba853c62698fb7a7a9c8fb3c90"} Jul 15 15:05:37.953033 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.952878 2567 scope.go:117] "RemoveContainer" containerID="f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5" Jul 15 15:05:37.963003 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.962988 2567 scope.go:117] "RemoveContainer" containerID="f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5" Jul 15 15:05:37.963257 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:05:37.963241 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5\": container with ID starting with f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5 not found: ID does not exist" containerID="f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5" Jul 15 15:05:37.963312 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.963264 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5"} err="failed to get container status \"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5\": rpc error: code = NotFound desc = could not find container \"f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5\": container with ID starting with f778c206c47d82b5699dbf017b1dd127907ef375a96c977f1abd4ec42217d8c5 not found: ID does not exist" Jul 15 15:05:37.974556 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.974532 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:05:37.979179 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:37.979158 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c3abd0e-7fdf8798c5-8swxf"] Jul 15 15:05:38.010946 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:38.010924 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d810a05-203e-4b5e-8733-2cc1de1cb698-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:38.010946 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:38.010944 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d810a05-203e-4b5e-8733-2cc1de1cb698-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:38.011102 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:38.010956 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8tb2\" (UniqueName: \"kubernetes.io/projected/5d810a05-203e-4b5e-8733-2cc1de1cb698-kube-api-access-m8tb2\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:38.165043 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:38.164967 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d810a05-203e-4b5e-8733-2cc1de1cb698" path="/var/lib/kubelet/pods/5d810a05-203e-4b5e-8733-2cc1de1cb698/volumes" Jul 15 15:05:52.316243 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:52.316209 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:05:52.316685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:52.316406 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" containerID="cri-o://ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45" gracePeriod=30 Jul 15 15:05:54.665875 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:54.665834 2567 patch_prober.go:28] interesting pod/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:05:54.666270 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:54.665907 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:05:57.574665 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.574638 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:05:57.679481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679443 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679518 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679536 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679570 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679591 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.679685 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679672 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffnzr\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.680007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679722 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.680007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679748 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert\") pod \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\" (UID: \"092a9570-9b25-4a19-aed4-2a23ee6e45ca\") " Jul 15 15:05:57.680007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679749 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:05:57.680007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679881 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:05:57.680007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.679937 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:05:57.680295 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.680131 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.680295 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.680150 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.680295 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.680162 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.680295 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.680234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:05:57.680515 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.680495 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:05:57.682075 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.682039 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:05:57.682183 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.682091 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:05:57.682297 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.682280 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:05:57.682350 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.682296 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr" (OuterVolumeSpecName: "kube-api-access-ffnzr") pod "092a9570-9b25-4a19-aed4-2a23ee6e45ca" (UID: "092a9570-9b25-4a19-aed4-2a23ee6e45ca"). InnerVolumeSpecName "kube-api-access-ffnzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:05:57.780859 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780826 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffnzr\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-kube-api-access-ffnzr\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.780859 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780853 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.780859 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780863 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.781127 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780873 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.781127 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780885 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:57.781127 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:57.780893 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092a9570-9b25-4a19-aed4-2a23ee6e45ca-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:05:58.060806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.060768 2567 generic.go:356] "Generic (PLEG): container finished" podID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerID="ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45" exitCode=0 Jul 15 15:05:58.060806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.060811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerDied","Data":"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45"} Jul 15 15:05:58.061094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.060821 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" Jul 15 15:05:58.061094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.060859 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44" event={"ID":"092a9570-9b25-4a19-aed4-2a23ee6e45ca","Type":"ContainerDied","Data":"92c0825e84563b293f8b2a3cdb25af68f7549e87a5db52e3eced98884a528ffb"} Jul 15 15:05:58.061094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.060882 2567 scope.go:117] "RemoveContainer" containerID="ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45" Jul 15 15:05:58.070698 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.070673 2567 scope.go:117] "RemoveContainer" containerID="3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252" Jul 15 15:05:58.079598 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.079577 2567 scope.go:117] "RemoveContainer" containerID="ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45" Jul 15 15:05:58.079864 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:05:58.079845 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45\": container with ID starting with ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45 not found: ID does not exist" containerID="ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45" Jul 15 15:05:58.079933 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.079878 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45"} err="failed to get container status \"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45\": rpc error: code = NotFound desc = could not find container \"ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45\": container with ID starting with ecc475f9ea71cbb75a715afc16b0f1b0a3bd4f4047bf83ef929354d560aadd45 not found: ID does not exist" Jul 15 15:05:58.079933 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.079905 2567 scope.go:117] "RemoveContainer" containerID="3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252" Jul 15 15:05:58.080217 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:05:58.080199 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252\": container with ID starting with 3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252 not found: ID does not exist" containerID="3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252" Jul 15 15:05:58.080269 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.080222 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252"} err="failed to get container status \"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252\": rpc error: code = NotFound desc = could not find container \"3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252\": container with ID starting with 3fa119c933537b2dbb7a820f3a128e3948e2a7438dadbe8165c9c19667d75252 not found: ID does not exist" Jul 15 15:05:58.085830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.085807 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:05:58.091687 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.091656 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9c3abd0e-openshift-default-67d5797d8c-pwb44"] Jul 15 15:05:58.165123 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:05:58.165088 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" path="/var/lib/kubelet/pods/092a9570-9b25-4a19-aed4-2a23ee6e45ca/volumes" Jul 15 15:07:30.677597 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.677556 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:07:30.678169 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678146 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d810a05-203e-4b5e-8733-2cc1de1cb698" containerName="maas-api" Jul 15 15:07:30.678169 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678168 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d810a05-203e-4b5e-8733-2cc1de1cb698" containerName="maas-api" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678201 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678209 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678215 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678270 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d810a05-203e-4b5e-8733-2cc1de1cb698" containerName="maas-api" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678279 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.678335 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.678285 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="092a9570-9b25-4a19-aed4-2a23ee6e45ca" containerName="istio-proxy" Jul 15 15:07:30.681354 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.681336 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.683219 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.683198 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88acceda-openshift-default-dockercfg-z7p8d\"" Jul 15 15:07:30.691521 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691647 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691647 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhgd\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691647 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691792 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691792 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691792 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691934 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.691934 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.691855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.699213 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.699189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:07:30.792874 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhgd\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.792997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793086 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793032 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793350 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793502 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793833 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793639 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.793981 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.793957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.795566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.795539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.795660 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.795634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.801020 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.800999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.801156 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.801131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhgd\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd\") pod \"e2e-mt-88acceda-openshift-default-5b598c6b79-lld68\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:30.992352 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:30.992244 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:31.133886 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.133857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:07:31.134038 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:07:31.133977 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bc1baf_e0a0_4bdc_8a4e_ac0010bbeeb9.slice/crio-005a5b73e968efb1b158eeac45319fd36c730d703af1df4e17f197a8fd8d209e WatchSource:0}: Error finding container 005a5b73e968efb1b158eeac45319fd36c730d703af1df4e17f197a8fd8d209e: Status 404 returned error can't find the container with id 005a5b73e968efb1b158eeac45319fd36c730d703af1df4e17f197a8fd8d209e Jul 15 15:07:31.137595 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.137561 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:31.137681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.137637 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:31.137732 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.137679 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:31.566333 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.566294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerStarted","Data":"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99"} Jul 15 15:07:31.566507 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.566351 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerStarted","Data":"005a5b73e968efb1b158eeac45319fd36c730d703af1df4e17f197a8fd8d209e"} Jul 15 15:07:31.627734 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.627673 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podStartSLOduration=1.627655833 podStartE2EDuration="1.627655833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:07:31.626312581 +0000 UTC m=+2022.035931860" watchObservedRunningTime="2026-07-15 15:07:31.627655833 +0000 UTC m=+2022.037275112" Jul 15 15:07:31.992652 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:31.992564 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:32.997907 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:32.997874 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:33.579461 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.579428 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:33.580537 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.580515 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:33.684582 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.684543 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:33.688731 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.688707 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.690529 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.690509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-88acceda-serving-cert\"" Jul 15 15:07:33.699824 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.699741 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:33.721656 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.721623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.721806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.721663 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc4b\" (UniqueName: \"kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.721806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.721701 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.822369 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.822331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.822536 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.822375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc4b\" (UniqueName: \"kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.822536 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.822421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.822536 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:07:33.822485 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-serving-cert: secret "maas-api-e2e-mt-88acceda-serving-cert" not found Jul 15 15:07:33.822726 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:07:33.822551 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls podName:d748a209-5e89-4f4f-80e4-a6478b44a97c nodeName:}" failed. No retries permitted until 2026-07-15 15:07:34.322529487 +0000 UTC m=+2024.732148743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls") pod "maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" (UID: "d748a209-5e89-4f4f-80e4-a6478b44a97c") : secret "maas-api-e2e-mt-88acceda-serving-cert" not found Jul 15 15:07:33.823087 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.823052 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:33.833394 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:33.833317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc4b\" (UniqueName: \"kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:34.327840 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:34.327803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:34.328373 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:07:34.327938 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-serving-cert: secret "maas-api-e2e-mt-88acceda-serving-cert" not found Jul 15 15:07:34.328373 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:07:34.328025 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls podName:d748a209-5e89-4f4f-80e4-a6478b44a97c nodeName:}" failed. No retries permitted until 2026-07-15 15:07:35.327987321 +0000 UTC m=+2025.737606578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls") pod "maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" (UID: "d748a209-5e89-4f4f-80e4-a6478b44a97c") : secret "maas-api-e2e-mt-88acceda-serving-cert" not found Jul 15 15:07:35.338751 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:35.338716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:35.341607 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:35.341583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"maas-api-e2e-mt-88acceda-76558fd79f-sfkzg\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:35.502568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:35.502530 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:35.647407 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:35.647379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:35.649770 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:07:35.649741 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd748a209_5e89_4f4f_80e4_a6478b44a97c.slice/crio-2d576c8f614f0dce9ba34092379ac1a7d15d34ccd74e3fe434d2625c02c24496 WatchSource:0}: Error finding container 2d576c8f614f0dce9ba34092379ac1a7d15d34ccd74e3fe434d2625c02c24496: Status 404 returned error can't find the container with id 2d576c8f614f0dce9ba34092379ac1a7d15d34ccd74e3fe434d2625c02c24496 Jul 15 15:07:35.651649 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:35.651630 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 15:07:36.598002 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:36.597969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" event={"ID":"d748a209-5e89-4f4f-80e4-a6478b44a97c","Type":"ContainerStarted","Data":"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d"} Jul 15 15:07:36.598475 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:36.598016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" event={"ID":"d748a209-5e89-4f4f-80e4-a6478b44a97c","Type":"ContainerStarted","Data":"2d576c8f614f0dce9ba34092379ac1a7d15d34ccd74e3fe434d2625c02c24496"} Jul 15 15:07:36.598475 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:36.598046 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:36.615681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:36.615611 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" podStartSLOduration=3.158857439 podStartE2EDuration="3.61559138s" totalImagesPullingTime="456.733941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:07:36.615263502 +0000 UTC m=+2027.024882783" watchObservedRunningTime="2026-07-15 15:07:36.61559138 +0000 UTC m=+2027.025210658" Jul 15 15:07:42.609213 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:42.609171 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:46.659670 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.659641 2567 generic.go:356] "Generic (PLEG): container finished" podID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerID="dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99" exitCode=0 Jul 15 15:07:46.660103 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.659710 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerDied","Data":"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99"} Jul 15 15:07:46.660157 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.660142 2567 scope.go:117] "RemoveContainer" containerID="dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99" Jul 15 15:07:46.661245 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.661218 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:46.661362 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.661301 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:46.661362 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:46.661345 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:07:47.667522 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:47.667491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerStarted","Data":"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c"} Jul 15 15:07:47.992572 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:47.992484 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:47.993652 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:47.993625 2567 patch_prober.go:28] interesting pod/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 15 15:07:47.993779 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:47.993671 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 15 15:07:48.579876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:48.579823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:48.993662 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:48.993571 2567 patch_prober.go:28] interesting pod/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 15 15:07:48.993662 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:48.993622 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 15 15:07:49.993473 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:49.993440 2567 patch_prober.go:28] interesting pod/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 15 15:07:49.993639 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:49.993487 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 15 15:07:50.993314 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:50.993277 2567 patch_prober.go:28] interesting pod/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 15 15:07:50.993672 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:50.993332 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 15 15:07:51.996332 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:51.996300 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:51.997416 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:51.997396 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:07:56.294653 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.294612 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:56.295158 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.294846 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" podUID="d748a209-5e89-4f4f-80e4-a6478b44a97c" containerName="maas-api" containerID="cri-o://bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d" gracePeriod=30 Jul 15 15:07:56.545128 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.545031 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:56.633352 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.633320 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca\") pod \"d748a209-5e89-4f4f-80e4-a6478b44a97c\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " Jul 15 15:07:56.633352 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.633357 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") pod \"d748a209-5e89-4f4f-80e4-a6478b44a97c\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " Jul 15 15:07:56.633590 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.633405 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhc4b\" (UniqueName: \"kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b\") pod \"d748a209-5e89-4f4f-80e4-a6478b44a97c\" (UID: \"d748a209-5e89-4f4f-80e4-a6478b44a97c\") " Jul 15 15:07:56.633741 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.633717 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d748a209-5e89-4f4f-80e4-a6478b44a97c" (UID: "d748a209-5e89-4f4f-80e4-a6478b44a97c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:07:56.635468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.635434 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d748a209-5e89-4f4f-80e4-a6478b44a97c" (UID: "d748a209-5e89-4f4f-80e4-a6478b44a97c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:07:56.635573 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.635530 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b" (OuterVolumeSpecName: "kube-api-access-dhc4b") pod "d748a209-5e89-4f4f-80e4-a6478b44a97c" (UID: "d748a209-5e89-4f4f-80e4-a6478b44a97c"). InnerVolumeSpecName "kube-api-access-dhc4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:07:56.717611 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.717582 2567 generic.go:356] "Generic (PLEG): container finished" podID="d748a209-5e89-4f4f-80e4-a6478b44a97c" containerID="bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d" exitCode=0 Jul 15 15:07:56.717723 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.717627 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" Jul 15 15:07:56.717723 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.717658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" event={"ID":"d748a209-5e89-4f4f-80e4-a6478b44a97c","Type":"ContainerDied","Data":"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d"} Jul 15 15:07:56.717723 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.717707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg" event={"ID":"d748a209-5e89-4f4f-80e4-a6478b44a97c","Type":"ContainerDied","Data":"2d576c8f614f0dce9ba34092379ac1a7d15d34ccd74e3fe434d2625c02c24496"} Jul 15 15:07:56.717848 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.717729 2567 scope.go:117] "RemoveContainer" containerID="bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d" Jul 15 15:07:56.727321 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.727304 2567 scope.go:117] "RemoveContainer" containerID="bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d" Jul 15 15:07:56.727566 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:07:56.727548 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d\": container with ID starting with bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d not found: ID does not exist" containerID="bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d" Jul 15 15:07:56.727636 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.727578 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d"} err="failed to get container status \"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d\": rpc error: code = NotFound desc = could not find container \"bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d\": container with ID starting with bb913f87ebfb7ce4ddfe8ae0a0f9e628a15761604d6288c216d79efab437c00d not found: ID does not exist" Jul 15 15:07:56.734045 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.734025 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d748a209-5e89-4f4f-80e4-a6478b44a97c-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:07:56.734045 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.734045 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d748a209-5e89-4f4f-80e4-a6478b44a97c-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:07:56.734180 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.734067 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc4b\" (UniqueName: \"kubernetes.io/projected/d748a209-5e89-4f4f-80e4-a6478b44a97c-kube-api-access-dhc4b\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:07:56.739838 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.739816 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:56.744154 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:56.744128 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88acceda-76558fd79f-sfkzg"] Jul 15 15:07:58.164431 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:07:58.164390 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d748a209-5e89-4f4f-80e4-a6478b44a97c" path="/var/lib/kubelet/pods/d748a209-5e89-4f4f-80e4-a6478b44a97c/volumes" Jul 15 15:08:11.545194 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:11.545111 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:08:11.545748 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:11.545378 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" containerID="cri-o://227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c" gracePeriod=30 Jul 15 15:08:16.812886 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.812855 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:08:16.829149 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.829044 2567 generic.go:356] "Generic (PLEG): container finished" podID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerID="227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c" exitCode=0 Jul 15 15:08:16.829149 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.829135 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" Jul 15 15:08:16.829317 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.829128 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerDied","Data":"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c"} Jul 15 15:08:16.829317 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.829299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68" event={"ID":"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9","Type":"ContainerDied","Data":"005a5b73e968efb1b158eeac45319fd36c730d703af1df4e17f197a8fd8d209e"} Jul 15 15:08:16.829391 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.829332 2567 scope.go:117] "RemoveContainer" containerID="227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c" Jul 15 15:08:16.844451 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.844427 2567 scope.go:117] "RemoveContainer" containerID="dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99" Jul 15 15:08:16.854344 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.854321 2567 scope.go:117] "RemoveContainer" containerID="227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c" Jul 15 15:08:16.854577 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:08:16.854558 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c\": container with ID starting with 227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c not found: ID does not exist" containerID="227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c" Jul 15 15:08:16.854632 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.854586 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c"} err="failed to get container status \"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c\": rpc error: code = NotFound desc = could not find container \"227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c\": container with ID starting with 227d937b23fe667b2ffffa28d184d79b220cc3f6287f892a501c104520d2977c not found: ID does not exist" Jul 15 15:08:16.854632 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.854604 2567 scope.go:117] "RemoveContainer" containerID="dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99" Jul 15 15:08:16.854862 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:08:16.854846 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99\": container with ID starting with dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99 not found: ID does not exist" containerID="dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99" Jul 15 15:08:16.854910 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.854866 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99"} err="failed to get container status \"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99\": rpc error: code = NotFound desc = could not find container \"dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99\": container with ID starting with dedf18ae7655ed260b181a6cd23a9c5328b7d558ff36bfb15a3dabdd3b77de99 not found: ID does not exist" Jul 15 15:08:16.912885 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.912855 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.912885 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.912889 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913111 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.912909 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913111 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.912947 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913111 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.912985 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snhgd\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913111 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913028 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913111 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913084 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913377 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:08:16.913377 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913200 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913377 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token\") pod \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\" (UID: \"53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9\") " Jul 15 15:08:16.913539 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:08:16.913539 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913457 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:08:16.913646 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913620 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:16.913646 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913640 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:16.913816 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913654 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:16.913816 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913750 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:08:16.913816 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.913781 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:08:16.915266 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.915241 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd" (OuterVolumeSpecName: "kube-api-access-snhgd") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "kube-api-access-snhgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:08:16.915373 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.915295 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:08:16.915492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.915473 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:08:16.915575 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:16.915558 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" (UID: "53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:08:17.014577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014533 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.014577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014571 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.014577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014580 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.014577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014589 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhgd\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-kube-api-access-snhgd\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.014577 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014598 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.014877 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.014606 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:08:17.151827 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.151797 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:08:17.159137 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:17.159112 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88acceda-openshift-default-5b598c6b79-lld68"] Jul 15 15:08:18.164784 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:18.164752 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" path="/var/lib/kubelet/pods/53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9/volumes" Jul 15 15:08:31.707810 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.707774 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:08:31.708403 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708383 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:31.708468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708407 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:31.708468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708421 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d748a209-5e89-4f4f-80e4-a6478b44a97c" containerName="maas-api" Jul 15 15:08:31.708468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708429 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d748a209-5e89-4f4f-80e4-a6478b44a97c" containerName="maas-api" Jul 15 15:08:31.708589 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708537 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:31.708589 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708552 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d748a209-5e89-4f4f-80e4-a6478b44a97c" containerName="maas-api" Jul 15 15:08:31.708589 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.708562 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:31.711672 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.711650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.713851 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.713823 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ca6c4f33-openshift-default-dockercfg-f95cv\"" Jul 15 15:08:31.721178 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.721153 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:08:31.848881 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.848844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849082 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.848913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849082 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.848943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849082 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.848967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcfb\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849082 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.849004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849082 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.849030 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849277 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.849148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849277 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.849186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.849277 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.849225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.949991 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.949955 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950192 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950000 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950192 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950192 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950192 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950192 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950457 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950457 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950457 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950281 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcfb\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950457 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950402 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950671 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950694 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.950829 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.950809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.951019 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.951002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.952440 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.952413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.952676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.952657 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.961189 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.961115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:31.961189 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:31.961145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcfb\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb\") pod \"e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:32.024246 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.024211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:32.177329 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.177299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:08:32.178903 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:08:32.178880 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73dab44_c2cb_42bf_9c82_511d630c7c6e.slice/crio-d85660da5fafaba472ae54fe3c1660bada80106ae94aa8f086e393c00551c6a5 WatchSource:0}: Error finding container d85660da5fafaba472ae54fe3c1660bada80106ae94aa8f086e393c00551c6a5: Status 404 returned error can't find the container with id d85660da5fafaba472ae54fe3c1660bada80106ae94aa8f086e393c00551c6a5 Jul 15 15:08:32.180952 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.180921 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:32.181039 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.180997 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:32.181125 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.181040 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:32.921807 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.921771 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerStarted","Data":"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b"} Jul 15 15:08:32.922235 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.921822 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerStarted","Data":"d85660da5fafaba472ae54fe3c1660bada80106ae94aa8f086e393c00551c6a5"} Jul 15 15:08:32.940755 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:32.940700 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podStartSLOduration=1.9406810129999998 podStartE2EDuration="1.940681013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:08:32.939955215 +0000 UTC m=+2083.349574493" watchObservedRunningTime="2026-07-15 15:08:32.940681013 +0000 UTC m=+2083.350300292" Jul 15 15:08:33.025013 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:33.024982 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:33.030559 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:33.030531 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:33.927178 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:33.927143 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:33.928029 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:33.927992 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:34.988599 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:34.988566 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:08:34.989561 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:34.989533 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:34.989561 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:34.989559 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bc1baf-e0a0-4bdc-8a4e-ac0010bbeeb9" containerName="istio-proxy" Jul 15 15:08:34.992718 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:34.992698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:34.994522 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:34.994502 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ca6c4f33-serving-cert\"" Jul 15 15:08:35.004977 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.004952 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:08:35.082944 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.082915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.083170 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.082959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.083170 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.083041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkvt\" (UniqueName: \"kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.183819 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.183786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkvt\" (UniqueName: \"kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.184044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.183855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.184044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.183885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.184535 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.184509 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.186559 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.186537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.192982 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.192956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkvt\" (UniqueName: \"kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt\") pod \"maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.307757 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.307728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.443359 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.443328 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:08:35.445787 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:08:35.445755 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c41496f_0231_40c0_8097_ea23dc60e72b.slice/crio-ab6bb93867c81eba7ce06dbb870417084d9d62ae6168f1f6072d40c3da1b2779 WatchSource:0}: Error finding container ab6bb93867c81eba7ce06dbb870417084d9d62ae6168f1f6072d40c3da1b2779: Status 404 returned error can't find the container with id ab6bb93867c81eba7ce06dbb870417084d9d62ae6168f1f6072d40c3da1b2779 Jul 15 15:08:35.941313 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.941271 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" event={"ID":"8c41496f-0231-40c0-8097-ea23dc60e72b","Type":"ContainerStarted","Data":"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2"} Jul 15 15:08:35.941480 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.941464 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:35.941540 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.941505 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" event={"ID":"8c41496f-0231-40c0-8097-ea23dc60e72b","Type":"ContainerStarted","Data":"ab6bb93867c81eba7ce06dbb870417084d9d62ae6168f1f6072d40c3da1b2779"} Jul 15 15:08:35.962068 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:35.962019 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" podStartSLOduration=1.564014442 podStartE2EDuration="1.962003648s" totalImagesPullingTime="397.989206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:08:35.95902652 +0000 UTC m=+2086.368645797" watchObservedRunningTime="2026-07-15 15:08:35.962003648 +0000 UTC m=+2086.371622961" Jul 15 15:08:41.958705 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:41.958669 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:08:47.007951 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.007915 2567 generic.go:356] "Generic (PLEG): container finished" podID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerID="dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b" exitCode=0 Jul 15 15:08:47.008417 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.007998 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerDied","Data":"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b"} Jul 15 15:08:47.008417 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.008414 2567 scope.go:117] "RemoveContainer" containerID="dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b" Jul 15 15:08:47.009487 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.009465 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:47.009543 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.009528 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:47.009584 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:47.009565 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:08:48.015414 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:48.015380 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerStarted","Data":"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658"} Jul 15 15:08:48.025279 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:48.025256 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:48.026449 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:48.026429 2567 patch_prober.go:28] interesting pod/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 15 15:08:48.026506 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:48.026476 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 15 15:08:48.927624 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:48.927594 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:49.024996 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:49.024965 2567 patch_prober.go:28] interesting pod/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 15 15:08:49.025432 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:49.025017 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 15 15:08:50.025392 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:50.025354 2567 patch_prober.go:28] interesting pod/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 15 15:08:50.025776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:50.025411 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 15 15:08:50.197976 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:50.197942 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:08:50.205520 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:50.205499 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:08:51.025284 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:51.025250 2567 patch_prober.go:28] interesting pod/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 15 15:08:51.025470 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:51.025319 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 15 15:08:52.028356 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:52.028323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:08:52.029451 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:08:52.029434 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:09:23.884587 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:23.884549 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:09:23.885254 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:23.884781 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" podUID="8c41496f-0231-40c0-8097-ea23dc60e72b" containerName="maas-api" containerID="cri-o://b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2" gracePeriod=30 Jul 15 15:09:24.141861 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.141790 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:09:24.220863 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.220824 2567 generic.go:356] "Generic (PLEG): container finished" podID="8c41496f-0231-40c0-8097-ea23dc60e72b" containerID="b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2" exitCode=0 Jul 15 15:09:24.221044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.220867 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" Jul 15 15:09:24.221044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.220909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" event={"ID":"8c41496f-0231-40c0-8097-ea23dc60e72b","Type":"ContainerDied","Data":"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2"} Jul 15 15:09:24.221044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.220963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s" event={"ID":"8c41496f-0231-40c0-8097-ea23dc60e72b","Type":"ContainerDied","Data":"ab6bb93867c81eba7ce06dbb870417084d9d62ae6168f1f6072d40c3da1b2779"} Jul 15 15:09:24.221044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.220991 2567 scope.go:117] "RemoveContainer" containerID="b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2" Jul 15 15:09:24.230295 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.230233 2567 scope.go:117] "RemoveContainer" containerID="b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2" Jul 15 15:09:24.230556 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:09:24.230531 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2\": container with ID starting with b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2 not found: ID does not exist" containerID="b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2" Jul 15 15:09:24.230676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.230564 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2"} err="failed to get container status \"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2\": rpc error: code = NotFound desc = could not find container \"b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2\": container with ID starting with b6799e17cbd0e5b28e0c6d2dbe555b5b813994f405685a54bd7c4e35841795f2 not found: ID does not exist" Jul 15 15:09:24.252970 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.252948 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca\") pod \"8c41496f-0231-40c0-8097-ea23dc60e72b\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " Jul 15 15:09:24.253101 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.252998 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls\") pod \"8c41496f-0231-40c0-8097-ea23dc60e72b\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " Jul 15 15:09:24.253210 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.253188 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krkvt\" (UniqueName: \"kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt\") pod \"8c41496f-0231-40c0-8097-ea23dc60e72b\" (UID: \"8c41496f-0231-40c0-8097-ea23dc60e72b\") " Jul 15 15:09:24.253321 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.253300 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c41496f-0231-40c0-8097-ea23dc60e72b" (UID: "8c41496f-0231-40c0-8097-ea23dc60e72b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:09:24.253533 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.253517 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c41496f-0231-40c0-8097-ea23dc60e72b-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:09:24.254903 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.254880 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c41496f-0231-40c0-8097-ea23dc60e72b" (UID: "8c41496f-0231-40c0-8097-ea23dc60e72b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:09:24.255144 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.255123 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt" (OuterVolumeSpecName: "kube-api-access-krkvt") pod "8c41496f-0231-40c0-8097-ea23dc60e72b" (UID: "8c41496f-0231-40c0-8097-ea23dc60e72b"). InnerVolumeSpecName "kube-api-access-krkvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:09:24.354797 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.354769 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c41496f-0231-40c0-8097-ea23dc60e72b-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:09:24.354797 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.354797 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krkvt\" (UniqueName: \"kubernetes.io/projected/8c41496f-0231-40c0-8097-ea23dc60e72b-kube-api-access-krkvt\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:09:24.543254 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.543221 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:09:24.547554 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:24.547532 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca6c4f33-645ff8fccf-rfj8s"] Jul 15 15:09:26.164828 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:26.164793 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c41496f-0231-40c0-8097-ea23dc60e72b" path="/var/lib/kubelet/pods/8c41496f-0231-40c0-8097-ea23dc60e72b/volumes" Jul 15 15:09:50.308251 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:09:50.308175 2567 scope.go:117] "RemoveContainer" containerID="03688b4ce09138fc1054f14ca5806518966944250c56dd3409d94d55f50592e5" Jul 15 15:10:10.724046 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:10.724015 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:10:10.724516 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:10.724266 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" containerID="cri-o://fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658" gracePeriod=30 Jul 15 15:10:16.002802 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.002774 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:10:16.118466 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118421 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118652 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118487 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118652 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118507 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118773 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118666 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118773 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118737 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:10:16.118773 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118752 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:10:16.118773 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118770 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118984 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118778 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:10:16.118984 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118822 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118984 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118881 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fcfb\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118984 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118917 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.118984 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.118949 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket\") pod \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\" (UID: \"c73dab44-c2cb-42bf-9c82-511d630c7c6e\") " Jul 15 15:10:16.119337 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.119233 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.119337 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.119249 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.119337 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.119262 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.119337 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.119229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:10:16.119604 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.119584 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:10:16.121019 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.120987 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:10:16.121136 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.120998 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:10:16.121136 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.121041 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:10:16.121325 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.121304 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb" (OuterVolumeSpecName: "kube-api-access-4fcfb") pod "c73dab44-c2cb-42bf-9c82-511d630c7c6e" (UID: "c73dab44-c2cb-42bf-9c82-511d630c7c6e"). InnerVolumeSpecName "kube-api-access-4fcfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:10:16.219948 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219921 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.219948 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219944 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.219948 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219952 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.220207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219972 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcfb\" (UniqueName: \"kubernetes.io/projected/c73dab44-c2cb-42bf-9c82-511d630c7c6e-kube-api-access-4fcfb\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.220207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219981 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73dab44-c2cb-42bf-9c82-511d630c7c6e-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.220207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.219989 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73dab44-c2cb-42bf-9c82-511d630c7c6e-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:10:16.515497 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.515463 2567 generic.go:356] "Generic (PLEG): container finished" podID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerID="fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658" exitCode=0 Jul 15 15:10:16.515676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.515510 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" Jul 15 15:10:16.515676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.515546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerDied","Data":"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658"} Jul 15 15:10:16.515676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.515601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp" event={"ID":"c73dab44-c2cb-42bf-9c82-511d630c7c6e","Type":"ContainerDied","Data":"d85660da5fafaba472ae54fe3c1660bada80106ae94aa8f086e393c00551c6a5"} Jul 15 15:10:16.515676 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.515623 2567 scope.go:117] "RemoveContainer" containerID="fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658" Jul 15 15:10:16.524762 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.524746 2567 scope.go:117] "RemoveContainer" containerID="dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b" Jul 15 15:10:16.533237 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.533219 2567 scope.go:117] "RemoveContainer" containerID="fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658" Jul 15 15:10:16.533489 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:10:16.533470 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658\": container with ID starting with fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658 not found: ID does not exist" containerID="fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658" Jul 15 15:10:16.533572 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.533501 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658"} err="failed to get container status \"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658\": rpc error: code = NotFound desc = could not find container \"fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658\": container with ID starting with fc052ee4d7d61a6c3b840dfa84026bb427a2255fac0c46ba16adbd900385c658 not found: ID does not exist" Jul 15 15:10:16.533572 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.533523 2567 scope.go:117] "RemoveContainer" containerID="dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b" Jul 15 15:10:16.533756 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:10:16.533738 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b\": container with ID starting with dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b not found: ID does not exist" containerID="dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b" Jul 15 15:10:16.533794 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.533762 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b"} err="failed to get container status \"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b\": rpc error: code = NotFound desc = could not find container \"dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b\": container with ID starting with dfb7138002594546d3ece44c7cc4a3c82e136fd7660d6f3da4712b6c9929eb3b not found: ID does not exist" Jul 15 15:10:16.534733 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.534717 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:10:16.539042 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:16.539021 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ca6c4f33-openshift-default-67c8759d79-9jzgp"] Jul 15 15:10:18.167224 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:10:18.167190 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" path="/var/lib/kubelet/pods/c73dab44-c2cb-42bf-9c82-511d630c7c6e/volumes" Jul 15 15:11:06.322495 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.322461 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:11:06.322985 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.322968 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323029 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.322988 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323029 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323011 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323029 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323020 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323142 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323050 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c41496f-0231-40c0-8097-ea23dc60e72b" containerName="maas-api" Jul 15 15:11:06.323142 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323075 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c41496f-0231-40c0-8097-ea23dc60e72b" containerName="maas-api" Jul 15 15:11:06.323207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323160 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323174 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73dab44-c2cb-42bf-9c82-511d630c7c6e" containerName="istio-proxy" Jul 15 15:11:06.323207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.323186 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c41496f-0231-40c0-8097-ea23dc60e72b" containerName="maas-api" Jul 15 15:11:06.326336 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.326315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.328132 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.328104 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e47fbc-openshift-default-dockercfg-bsmzx\"" Jul 15 15:11:06.338143 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.337978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:11:06.430906 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.430876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67f77\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431090 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.430927 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431090 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.430958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431090 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431090 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431291 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431291 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431291 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.431394 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.431294 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532126 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532286 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532286 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532286 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67f77\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532717 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.532983 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.533101 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.532972 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.533215 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.533181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.533313 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.533255 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.535038 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.535015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.535134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.535033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.541996 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.541971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.542417 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.542399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67f77\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77\") pod \"e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.638248 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.638181 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:06.783586 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.783545 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:11:06.785260 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:11:06.785180 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b58a5e2_641a_4590_b089_003068574aa5.slice/crio-14dada612b62d8f23534b46e09d31578ebf6eb588ffabbb025739a537fe82674 WatchSource:0}: Error finding container 14dada612b62d8f23534b46e09d31578ebf6eb588ffabbb025739a537fe82674: Status 404 returned error can't find the container with id 14dada612b62d8f23534b46e09d31578ebf6eb588ffabbb025739a537fe82674 Jul 15 15:11:06.787644 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.787617 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:06.787728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.787677 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:06.787728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:06.787702 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:07.794639 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:07.794591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerStarted","Data":"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737"} Jul 15 15:11:07.795164 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:07.794647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerStarted","Data":"14dada612b62d8f23534b46e09d31578ebf6eb588ffabbb025739a537fe82674"} Jul 15 15:11:07.813785 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:07.813741 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podStartSLOduration=1.813726535 podStartE2EDuration="1.813726535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:11:07.812657677 +0000 UTC m=+2238.222276957" watchObservedRunningTime="2026-07-15 15:11:07.813726535 +0000 UTC m=+2238.223345811" Jul 15 15:11:08.638565 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:08.638534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:08.643624 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:08.643600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:08.799831 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:08.799807 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:08.800782 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:08.800762 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:09.859638 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.859610 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:11:09.863149 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.863128 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:09.865655 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.865635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e47fbc-serving-cert\"" Jul 15 15:11:09.875951 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.875925 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:11:09.964572 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.964535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptrz\" (UniqueName: \"kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:09.964717 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.964594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:09.964717 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:09.964688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.065341 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.065311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.065486 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.065370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptrz\" (UniqueName: \"kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.065486 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.065421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.066015 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.065988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.067512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.067493 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.073225 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.073190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptrz\" (UniqueName: \"kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz\") pod \"maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.175049 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.174988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:10.521045 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.521022 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:11:10.523683 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:11:10.523659 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377787e2_1d08_49e2_8060_b5ccc7ba8833.slice/crio-144e636603192e1daafec1d3c8c86ef394488ab5b331101ee80d0df50dbb2624 WatchSource:0}: Error finding container 144e636603192e1daafec1d3c8c86ef394488ab5b331101ee80d0df50dbb2624: Status 404 returned error can't find the container with id 144e636603192e1daafec1d3c8c86ef394488ab5b331101ee80d0df50dbb2624 Jul 15 15:11:10.811604 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:10.811575 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" event={"ID":"377787e2-1d08-49e2-8060-b5ccc7ba8833","Type":"ContainerStarted","Data":"144e636603192e1daafec1d3c8c86ef394488ab5b331101ee80d0df50dbb2624"} Jul 15 15:11:11.818257 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:11.818220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" event={"ID":"377787e2-1d08-49e2-8060-b5ccc7ba8833","Type":"ContainerStarted","Data":"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3"} Jul 15 15:11:11.818695 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:11.818286 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:11.839497 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:11.839445 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" podStartSLOduration=2.374471743 podStartE2EDuration="2.839427939s" totalImagesPullingTime="464.956196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:11:11.836821554 +0000 UTC m=+2242.246440836" watchObservedRunningTime="2026-07-15 15:11:11.839427939 +0000 UTC m=+2242.249047218" Jul 15 15:11:17.829342 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:17.829309 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:11:24.842205 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:24.842166 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 15:11:24.842592 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:24.842224 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 15:11:25.901694 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.901663 2567 generic.go:356] "Generic (PLEG): container finished" podID="8b58a5e2-641a-4590-b089-003068574aa5" containerID="0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737" exitCode=0 Jul 15 15:11:25.902207 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.901735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerDied","Data":"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737"} Jul 15 15:11:25.902283 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.902231 2567 scope.go:117] "RemoveContainer" containerID="0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737" Jul 15 15:11:25.903201 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.903175 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:25.903311 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.903239 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:25.903311 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:25.903274 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:11:26.909301 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:26.909266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerStarted","Data":"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871"} Jul 15 15:11:27.638677 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:27.638645 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:27.639009 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:27.638985 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 15 15:11:27.639198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:27.639172 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 15 15:11:28.638883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:28.638845 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 15 15:11:28.639527 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:28.639494 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 15 15:11:29.639238 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:29.639191 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 15 15:11:29.639744 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:29.639277 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 15 15:11:30.639616 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:30.639576 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 15 15:11:30.640261 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:30.640227 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 15 15:11:31.642761 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:31.642724 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:31.643285 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:31.643068 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:11:31.643893 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:11:31.643871 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:12:42.283648 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.283535 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:12:42.284223 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.283749 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" podUID="377787e2-1d08-49e2-8060-b5ccc7ba8833" containerName="maas-api" containerID="cri-o://40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3" gracePeriod=30 Jul 15 15:12:42.530996 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.530964 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:12:42.618184 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.618151 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hptrz\" (UniqueName: \"kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz\") pod \"377787e2-1d08-49e2-8060-b5ccc7ba8833\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " Jul 15 15:12:42.618316 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.618255 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls\") pod \"377787e2-1d08-49e2-8060-b5ccc7ba8833\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " Jul 15 15:12:42.618316 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.618301 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca\") pod \"377787e2-1d08-49e2-8060-b5ccc7ba8833\" (UID: \"377787e2-1d08-49e2-8060-b5ccc7ba8833\") " Jul 15 15:12:42.618663 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.618638 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "377787e2-1d08-49e2-8060-b5ccc7ba8833" (UID: "377787e2-1d08-49e2-8060-b5ccc7ba8833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:12:42.620131 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.620112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "377787e2-1d08-49e2-8060-b5ccc7ba8833" (UID: "377787e2-1d08-49e2-8060-b5ccc7ba8833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:12:42.620233 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.620213 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz" (OuterVolumeSpecName: "kube-api-access-hptrz") pod "377787e2-1d08-49e2-8060-b5ccc7ba8833" (UID: "377787e2-1d08-49e2-8060-b5ccc7ba8833"). InnerVolumeSpecName "kube-api-access-hptrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:12:42.719584 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.719563 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377787e2-1d08-49e2-8060-b5ccc7ba8833-maas-api-tls\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:12:42.719584 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.719582 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377787e2-1d08-49e2-8060-b5ccc7ba8833-openshift-service-ca\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:12:42.719711 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:42.719594 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptrz\" (UniqueName: \"kubernetes.io/projected/377787e2-1d08-49e2-8060-b5ccc7ba8833-kube-api-access-hptrz\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:12:43.325702 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.325667 2567 generic.go:356] "Generic (PLEG): container finished" podID="377787e2-1d08-49e2-8060-b5ccc7ba8833" containerID="40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3" exitCode=0 Jul 15 15:12:43.326077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.325717 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" Jul 15 15:12:43.326077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.325750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" event={"ID":"377787e2-1d08-49e2-8060-b5ccc7ba8833","Type":"ContainerDied","Data":"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3"} Jul 15 15:12:43.326077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.325803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb" event={"ID":"377787e2-1d08-49e2-8060-b5ccc7ba8833","Type":"ContainerDied","Data":"144e636603192e1daafec1d3c8c86ef394488ab5b331101ee80d0df50dbb2624"} Jul 15 15:12:43.326077 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.325826 2567 scope.go:117] "RemoveContainer" containerID="40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3" Jul 15 15:12:43.338664 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.338642 2567 scope.go:117] "RemoveContainer" containerID="40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3" Jul 15 15:12:43.338947 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:12:43.338928 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3\": container with ID starting with 40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3 not found: ID does not exist" containerID="40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3" Jul 15 15:12:43.339000 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.338957 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3"} err="failed to get container status \"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3\": rpc error: code = NotFound desc = could not find container \"40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3\": container with ID starting with 40c6c42a1c88617ec4d527bfe8293b4d3f2bdf158b40eff023ab523432677ab3 not found: ID does not exist" Jul 15 15:12:43.350624 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.350604 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:12:43.358618 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:43.358601 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e47fbc-69fd6868ff-lpsbb"] Jul 15 15:12:44.164476 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:44.164449 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377787e2-1d08-49e2-8060-b5ccc7ba8833" path="/var/lib/kubelet/pods/377787e2-1d08-49e2-8060-b5ccc7ba8833/volumes" Jul 15 15:12:57.605114 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:57.605081 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:12:57.605486 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:12:57.605299 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" containerID="cri-o://d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871" gracePeriod=30 Jul 15 15:13:01.645767 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:01.645726 2567 patch_prober.go:28] interesting pod/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:13:01.646145 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:01.645800 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:13:02.861536 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.861512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:13:02.905490 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905402 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905490 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905439 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67f77\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905490 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905461 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905731 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905501 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905731 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905682 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905731 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905706 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:02.905889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905729 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905775 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.905889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905781 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:02.905889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905793 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:02.905889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905865 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.906172 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.905893 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo\") pod \"8b58a5e2-641a-4590-b089-003068574aa5\" (UID: \"8b58a5e2-641a-4590-b089-003068574aa5\") " Jul 15 15:13:02.906172 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906105 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:13:02.906284 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906223 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:02.906284 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906241 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58a5e2-641a-4590-b089-003068574aa5-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:02.906284 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906253 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:02.906284 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906265 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:02.906539 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.906515 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:02.908094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.908044 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:13:02.908296 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.908260 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77" (OuterVolumeSpecName: "kube-api-access-67f77") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "kube-api-access-67f77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:13:02.908404 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.908311 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:13:02.908404 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:02.908334 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b58a5e2-641a-4590-b089-003068574aa5" (UID: "8b58a5e2-641a-4590-b089-003068574aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:03.007010 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.006959 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:03.007010 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.007003 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:03.007010 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.007012 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58a5e2-641a-4590-b089-003068574aa5-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:03.007010 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.007020 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58a5e2-641a-4590-b089-003068574aa5-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:03.007010 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.007030 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67f77\" (UniqueName: \"kubernetes.io/projected/8b58a5e2-641a-4590-b089-003068574aa5-kube-api-access-67f77\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:13:03.437329 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.437292 2567 generic.go:356] "Generic (PLEG): container finished" podID="8b58a5e2-641a-4590-b089-003068574aa5" containerID="d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871" exitCode=0 Jul 15 15:13:03.437568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.437343 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" Jul 15 15:13:03.437568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.437375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerDied","Data":"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871"} Jul 15 15:13:03.437568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.437432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28" event={"ID":"8b58a5e2-641a-4590-b089-003068574aa5","Type":"ContainerDied","Data":"14dada612b62d8f23534b46e09d31578ebf6eb588ffabbb025739a537fe82674"} Jul 15 15:13:03.437568 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.437456 2567 scope.go:117] "RemoveContainer" containerID="d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871" Jul 15 15:13:03.447735 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.447717 2567 scope.go:117] "RemoveContainer" containerID="0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737" Jul 15 15:13:03.455939 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.455922 2567 scope.go:117] "RemoveContainer" containerID="d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871" Jul 15 15:13:03.456215 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:13:03.456199 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871\": container with ID starting with d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871 not found: ID does not exist" containerID="d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871" Jul 15 15:13:03.456271 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.456222 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871"} err="failed to get container status \"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871\": rpc error: code = NotFound desc = could not find container \"d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871\": container with ID starting with d5169f49469d86c09d903f787055e8167235f92335925bdda45e8478e4756871 not found: ID does not exist" Jul 15 15:13:03.456271 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.456239 2567 scope.go:117] "RemoveContainer" containerID="0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737" Jul 15 15:13:03.456489 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:13:03.456471 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737\": container with ID starting with 0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737 not found: ID does not exist" containerID="0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737" Jul 15 15:13:03.456536 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.456495 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737"} err="failed to get container status \"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737\": rpc error: code = NotFound desc = could not find container \"0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737\": container with ID starting with 0344a60657cce37ae021fd5585d4b2c3dc49b4c400a7c01d48eb8692e55df737 not found: ID does not exist" Jul 15 15:13:03.460269 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.460246 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:13:03.464862 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:03.464840 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e47fbc-openshift-default-67c5764699-qvd28"] Jul 15 15:13:04.164484 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:04.164450 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b58a5e2-641a-4590-b089-003068574aa5" path="/var/lib/kubelet/pods/8b58a5e2-641a-4590-b089-003068574aa5/volumes" Jul 15 15:13:24.631013 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.630980 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:13:24.632099 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632045 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.632099 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632097 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.632099 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632107 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632113 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632126 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377787e2-1d08-49e2-8060-b5ccc7ba8833" containerName="maas-api" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632132 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="377787e2-1d08-49e2-8060-b5ccc7ba8833" containerName="maas-api" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632205 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="377787e2-1d08-49e2-8060-b5ccc7ba8833" containerName="maas-api" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632214 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.632365 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.632222 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b58a5e2-641a-4590-b089-003068574aa5" containerName="istio-proxy" Jul 15 15:13:24.635071 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.635040 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.637161 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.637139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c7c00d-openshift-default-dockercfg-dhl64\"" Jul 15 15:13:24.645213 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.645175 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:13:24.705956 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.705920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706158 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.705965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706158 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706158 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706158 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmlcz\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706358 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706358 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706358 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.706358 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.706250 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807450 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807450 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807655 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807706 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807753 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807806 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807856 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807856 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.807856 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.808023 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.808023 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.807946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmlcz\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.808183 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.808136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.808372 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.808352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.808477 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.808456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.810307 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.810287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.810408 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.810321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.816047 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.816028 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmlcz\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.816047 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.816038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token\") pod \"e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:24.947749 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:24.947667 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:25.090547 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.090518 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:13:25.091893 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:13:25.091868 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ce0207_4167_49bd_8dad_b66485593761.slice/crio-3e71535724c4a2fcd4d9fb8263d99e9376c0660569e35c4745bc9e5fd7fbb3c8 WatchSource:0}: Error finding container 3e71535724c4a2fcd4d9fb8263d99e9376c0660569e35c4745bc9e5fd7fbb3c8: Status 404 returned error can't find the container with id 3e71535724c4a2fcd4d9fb8263d99e9376c0660569e35c4745bc9e5fd7fbb3c8 Jul 15 15:13:25.094072 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.094019 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:25.094177 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.094122 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:25.094177 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.094167 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:25.563442 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.563399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerStarted","Data":"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575"} Jul 15 15:13:25.563630 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.563462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerStarted","Data":"3e71535724c4a2fcd4d9fb8263d99e9376c0660569e35c4745bc9e5fd7fbb3c8"} Jul 15 15:13:25.589981 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.589932 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podStartSLOduration=1.589918123 podStartE2EDuration="1.589918123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:13:25.586597536 +0000 UTC m=+2375.996216812" watchObservedRunningTime="2026-07-15 15:13:25.589918123 +0000 UTC m=+2375.999537400" Jul 15 15:13:25.948000 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:25.947906 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:26.953678 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:26.953648 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:27.573712 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:27.573673 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:27.574500 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:27.574482 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:32.770123 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.770086 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:13:32.772712 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.772687 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.774896 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.774865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-345426-openshift-default-dockercfg-tbt66\"" Jul 15 15:13:32.785376 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.785354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:13:32.889754 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.889721 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.889754 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.889758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.889877 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62tp\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.889935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.889969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890007 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.890004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890254 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.890029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890254 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.890112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.890254 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.890173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990743 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990758 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.990942 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.990919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62tp\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991270 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991348 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991328 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991415 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991113 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991471 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991410 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991610 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991745 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.991891 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.991869 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.993345 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.993320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.993571 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.993553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.998816 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.998795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62tp\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:32.999021 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:32.999001 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token\") pod \"e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:33.085343 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.085307 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:33.230471 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.230430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:13:33.233001 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:13:33.232956 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feb76c4_4077_4523_b3a7_ee00dd9a56e0.slice/crio-276abd03212d19575e1237c1be130bb3f062bca35530783d6d1cff92761d73df WatchSource:0}: Error finding container 276abd03212d19575e1237c1be130bb3f062bca35530783d6d1cff92761d73df: Status 404 returned error can't find the container with id 276abd03212d19575e1237c1be130bb3f062bca35530783d6d1cff92761d73df Jul 15 15:13:33.236029 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.235975 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:33.236170 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.236153 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:33.236231 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.236202 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:33.607124 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.607087 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerStarted","Data":"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d"} Jul 15 15:13:33.607312 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.607138 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerStarted","Data":"276abd03212d19575e1237c1be130bb3f062bca35530783d6d1cff92761d73df"} Jul 15 15:13:33.627909 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:33.627858 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podStartSLOduration=1.6278431580000001 podStartE2EDuration="1.627843158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:13:33.6259384 +0000 UTC m=+2384.035557670" watchObservedRunningTime="2026-07-15 15:13:33.627843158 +0000 UTC m=+2384.037462436" Jul 15 15:13:34.085520 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:34.085484 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:34.090603 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:34.090577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:34.612433 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:34.612395 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:34.613323 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:34.613306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:40.649658 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.649569 2567 generic.go:356] "Generic (PLEG): container finished" podID="81ce0207-4167-49bd-8dad-b66485593761" containerID="646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575" exitCode=0 Jul 15 15:13:40.649658 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.649629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerDied","Data":"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575"} Jul 15 15:13:40.650184 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.649991 2567 scope.go:117] "RemoveContainer" containerID="646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575" Jul 15 15:13:40.651151 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.651121 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:40.651219 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.651205 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:40.651255 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:40.651245 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:41.679137 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:41.679100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerStarted","Data":"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb"} Jul 15 15:13:41.950195 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:41.950101 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:41.950681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:41.950657 2567 patch_prober.go:28] interesting pod/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 15 15:13:41.950867 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:41.950843 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 15 15:13:42.573905 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:42.573863 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:42.948450 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:42.948338 2567 patch_prober.go:28] interesting pod/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 15 15:13:42.948450 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:42.948428 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 15 15:13:43.953084 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:43.952466 2567 patch_prober.go:28] interesting pod/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 15 15:13:43.953084 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:43.952537 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 15 15:13:44.948337 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:44.948294 2567 patch_prober.go:28] interesting pod/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 15 15:13:44.948555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:44.948373 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 15 15:13:45.953629 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:45.953595 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:45.954716 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:45.954692 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:13:48.720866 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.720831 2567 generic.go:356] "Generic (PLEG): container finished" podID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerID="be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d" exitCode=0 Jul 15 15:13:48.721383 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.720899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerDied","Data":"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d"} Jul 15 15:13:48.721383 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.721317 2567 scope.go:117] "RemoveContainer" containerID="be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d" Jul 15 15:13:48.722349 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.722324 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:48.722408 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.722394 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:48.722445 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:48.722425 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:13:49.612330 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:49.612292 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:49.730008 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:49.729962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerStarted","Data":"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc"} Jul 15 15:13:50.086034 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:50.086002 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:50.086803 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:50.086781 2567 patch_prober.go:28] interesting pod/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 15 15:13:50.086865 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:50.086845 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 15 15:13:50.254212 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:50.253883 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:13:50.266520 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:50.266496 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:13:51.086594 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:51.086560 2567 patch_prober.go:28] interesting pod/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 15 15:13:51.086986 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:51.086615 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 15 15:13:52.086233 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:52.086193 2567 patch_prober.go:28] interesting pod/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 15 15:13:52.086419 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:52.086261 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 15 15:13:53.085926 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:53.085887 2567 patch_prober.go:28] interesting pod/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 15 15:13:53.086343 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:53.085960 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 15 15:13:54.089514 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:54.089485 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:54.089993 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:54.089905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:13:54.090684 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:13:54.090665 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:14:13.888594 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:13.888514 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:14:13.889151 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:13.888770 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" containerID="cri-o://f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb" gracePeriod=30 Jul 15 15:14:15.956677 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:15.956638 2567 patch_prober.go:28] interesting pod/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:14:15.957087 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:15.956695 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:14:19.139598 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.139567 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:14:19.243750 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243658 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.243750 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.243750 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243783 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243799 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243832 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243888 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243920 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244091 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.243964 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmlcz\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz\") pod \"81ce0207-4167-49bd-8dad-b66485593761\" (UID: \"81ce0207-4167-49bd-8dad-b66485593761\") " Jul 15 15:14:19.244424 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:19.244424 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244191 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:19.244424 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:19.244600 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244462 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.244600 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:14:19.244600 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244483 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.244600 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.244509 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.245261 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.245238 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data" (OuterVolumeSpecName: "istio-data") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:19.246138 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.246112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:19.246271 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.246251 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz" (OuterVolumeSpecName: "kube-api-access-dmlcz") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "kube-api-access-dmlcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:14:19.246420 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.246404 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:14:19.246480 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.246417 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token" (OuterVolumeSpecName: "istio-token") pod "81ce0207-4167-49bd-8dad-b66485593761" (UID: "81ce0207-4167-49bd-8dad-b66485593761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:14:19.345134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345100 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.345134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345128 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce0207-4167-49bd-8dad-b66485593761-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.345134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345137 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmlcz\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-kube-api-access-dmlcz\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.345134 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345146 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce0207-4167-49bd-8dad-b66485593761-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.345410 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345155 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce0207-4167-49bd-8dad-b66485593761-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.345410 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.345180 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce0207-4167-49bd-8dad-b66485593761-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:19.905606 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.905568 2567 generic.go:356] "Generic (PLEG): container finished" podID="81ce0207-4167-49bd-8dad-b66485593761" containerID="f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb" exitCode=0 Jul 15 15:14:19.905780 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.905630 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" Jul 15 15:14:19.905780 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.905656 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerDied","Data":"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb"} Jul 15 15:14:19.905780 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.905733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l" event={"ID":"81ce0207-4167-49bd-8dad-b66485593761","Type":"ContainerDied","Data":"3e71535724c4a2fcd4d9fb8263d99e9376c0660569e35c4745bc9e5fd7fbb3c8"} Jul 15 15:14:19.905780 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.905757 2567 scope.go:117] "RemoveContainer" containerID="f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb" Jul 15 15:14:19.915654 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.915635 2567 scope.go:117] "RemoveContainer" containerID="646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575" Jul 15 15:14:19.923820 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.923798 2567 scope.go:117] "RemoveContainer" containerID="f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb" Jul 15 15:14:19.924109 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:14:19.924089 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb\": container with ID starting with f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb not found: ID does not exist" containerID="f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb" Jul 15 15:14:19.924164 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.924117 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb"} err="failed to get container status \"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb\": rpc error: code = NotFound desc = could not find container \"f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb\": container with ID starting with f81903b0348802c5685ca406f71dd64dba46d6acdb1e9c34e94aa470c9e5c8bb not found: ID does not exist" Jul 15 15:14:19.924164 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.924134 2567 scope.go:117] "RemoveContainer" containerID="646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575" Jul 15 15:14:19.924366 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:14:19.924349 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575\": container with ID starting with 646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575 not found: ID does not exist" containerID="646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575" Jul 15 15:14:19.924414 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.924373 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575"} err="failed to get container status \"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575\": rpc error: code = NotFound desc = could not find container \"646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575\": container with ID starting with 646780ca786076bc273a2378d2106eb05ddcb7cc571dc0598326b3ee22576575 not found: ID does not exist" Jul 15 15:14:19.932128 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.932108 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:14:19.936440 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:19.936421 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c7c00d-openshift-default-cc4b4d8d5-mf28l"] Jul 15 15:14:20.164496 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:20.164412 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ce0207-4167-49bd-8dad-b66485593761" path="/var/lib/kubelet/pods/81ce0207-4167-49bd-8dad-b66485593761/volumes" Jul 15 15:14:40.051698 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.051661 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:14:40.052174 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.051895 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" containerID="cri-o://19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc" gracePeriod=30 Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.980698 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981283 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981299 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981319 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981328 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981431 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.984198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.981440 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ce0207-4167-49bd-8dad-b66485593761" containerName="istio-proxy" Jul 15 15:14:40.985371 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.985346 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:40.987857 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.987659 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0c81d7-openshift-default-dockercfg-q9vbv\"" Jul 15 15:14:40.999679 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:40.999656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:14:41.031686 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031655 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.031818 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf8h\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.031818 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.031935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.031935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.031935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.031899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.032070 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.032016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.032119 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.032048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.032189 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.032172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.132878 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.132840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.132878 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.132881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.132906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf8h\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133288 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133306 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133434 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133847 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133510 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.133847 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.133684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.134476 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.134447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.134935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.134913 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.135448 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.135421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.135943 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.135911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.142027 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.142000 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.142308 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.142291 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf8h\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h\") pod \"e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.298935 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.298863 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:41.437515 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.437486 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:14:41.439242 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:14:41.439205 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecd5961_d849_4980_93e3_55807fe7ea37.slice/crio-a3451fb7103082e233cff0846233ed0ca9d1c8243444629835ff53963f6d9425 WatchSource:0}: Error finding container a3451fb7103082e233cff0846233ed0ca9d1c8243444629835ff53963f6d9425: Status 404 returned error can't find the container with id a3451fb7103082e233cff0846233ed0ca9d1c8243444629835ff53963f6d9425 Jul 15 15:14:41.441548 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.441511 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:41.441639 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.441590 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:41.441694 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:41.441634 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:42.025562 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:42.025528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerStarted","Data":"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b"} Jul 15 15:14:42.025731 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:42.025584 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerStarted","Data":"a3451fb7103082e233cff0846233ed0ca9d1c8243444629835ff53963f6d9425"} Jul 15 15:14:42.046949 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:42.046907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podStartSLOduration=2.046892599 podStartE2EDuration="2.046892599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:14:42.045358442 +0000 UTC m=+2452.454977746" watchObservedRunningTime="2026-07-15 15:14:42.046892599 +0000 UTC m=+2452.456511877" Jul 15 15:14:42.299914 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:42.299884 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:42.304341 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:42.304319 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:43.032614 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:43.032579 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:43.033500 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:43.033476 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:45.317347 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.317283 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:14:45.370422 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370396 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370582 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370457 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370582 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370582 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370516 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k62tp\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370766 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370582 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370766 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370622 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370766 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370642 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370766 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370685 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.370766 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.370741 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo\") pod \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\" (UID: \"9feb76c4-4077-4523-b3a7-ee00dd9a56e0\") " Jul 15 15:14:45.371178 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.371139 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:45.371384 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.371162 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:45.371492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.371466 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:14:45.371492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.371275 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:45.371492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.371453 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:45.372787 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.372766 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp" (OuterVolumeSpecName: "kube-api-access-k62tp") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "kube-api-access-k62tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:14:45.372955 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.372936 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:14:45.373110 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.373089 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:14:45.373180 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.373100 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9feb76c4-4077-4523-b3a7-ee00dd9a56e0" (UID: "9feb76c4-4077-4523-b3a7-ee00dd9a56e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:14:45.472372 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472344 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472372 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472368 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472372 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472376 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472384 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472392 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472399 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472407 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472415 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62tp\" (UniqueName: \"kubernetes.io/projected/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-kube-api-access-k62tp\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:45.472555 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:45.472426 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9feb76c4-4077-4523-b3a7-ee00dd9a56e0-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:14:46.053385 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.053347 2567 generic.go:356] "Generic (PLEG): container finished" podID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerID="19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc" exitCode=0 Jul 15 15:14:46.053565 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.053396 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" Jul 15 15:14:46.053565 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.053434 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerDied","Data":"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc"} Jul 15 15:14:46.053565 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.053495 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8" event={"ID":"9feb76c4-4077-4523-b3a7-ee00dd9a56e0","Type":"ContainerDied","Data":"276abd03212d19575e1237c1be130bb3f062bca35530783d6d1cff92761d73df"} Jul 15 15:14:46.053565 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.053525 2567 scope.go:117] "RemoveContainer" containerID="19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc" Jul 15 15:14:46.063428 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.063405 2567 scope.go:117] "RemoveContainer" containerID="be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d" Jul 15 15:14:46.071529 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.071515 2567 scope.go:117] "RemoveContainer" containerID="19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc" Jul 15 15:14:46.071767 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:14:46.071748 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc\": container with ID starting with 19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc not found: ID does not exist" containerID="19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc" Jul 15 15:14:46.071829 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.071774 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc"} err="failed to get container status \"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc\": rpc error: code = NotFound desc = could not find container \"19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc\": container with ID starting with 19ef07e803223390070b7df7c4f757653dfd112ea88dbfe08dd6991a432c41fc not found: ID does not exist" Jul 15 15:14:46.071829 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.071792 2567 scope.go:117] "RemoveContainer" containerID="be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d" Jul 15 15:14:46.072072 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:14:46.072033 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d\": container with ID starting with be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d not found: ID does not exist" containerID="be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d" Jul 15 15:14:46.072141 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.072078 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d"} err="failed to get container status \"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d\": rpc error: code = NotFound desc = could not find container \"be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d\": container with ID starting with be3b1fda17763323a5905912846d3cadeee7556f3e6d0efa4c166d6a5d40f35d not found: ID does not exist" Jul 15 15:14:46.075025 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.075004 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:14:46.082313 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.082292 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-345426-openshift-default-577bb9bc77-sr4p8"] Jul 15 15:14:46.165868 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:46.165838 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" path="/var/lib/kubelet/pods/9feb76c4-4077-4523-b3a7-ee00dd9a56e0/volumes" Jul 15 15:14:49.099398 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.099255 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:14:49.099882 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.099818 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.099882 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.099837 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.099882 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.099854 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.099882 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.099863 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.100183 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.100007 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.100183 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.100020 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9feb76c4-4077-4523-b3a7-ee00dd9a56e0" containerName="istio-proxy" Jul 15 15:14:49.102321 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.102301 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.104491 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.104471 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b2aed6-openshift-default-dockercfg-qz7zp\"" Jul 15 15:14:49.115559 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.115537 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:14:49.207034 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.206999 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207236 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207318 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbg2\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207372 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207433 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207509 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207570 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207626 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207575 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.207681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.207629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308463 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbg2\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308775 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.308830 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.308801 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.309424 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.309409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.309545 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.309523 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.309725 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.309692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.309960 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.309937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.310164 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.310144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.311280 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.311258 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.311856 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.311838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.317018 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.316990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbg2\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.317122 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.317113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token\") pod \"e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.413993 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.413902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:49.569330 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.569301 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:14:49.570921 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:14:49.570895 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71d3c53_9955_4c0c_8638_305d015e8c7a.slice/crio-7eaec0b6c3433364e3d769ef982c6f4054fb43a4222a347a71461cee91bb498c WatchSource:0}: Error finding container 7eaec0b6c3433364e3d769ef982c6f4054fb43a4222a347a71461cee91bb498c: Status 404 returned error can't find the container with id 7eaec0b6c3433364e3d769ef982c6f4054fb43a4222a347a71461cee91bb498c Jul 15 15:14:49.573312 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.573276 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:49.573410 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.573384 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:49.573473 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:49.573428 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:50.079314 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:50.079277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerStarted","Data":"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1"} Jul 15 15:14:50.079533 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:50.079342 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerStarted","Data":"7eaec0b6c3433364e3d769ef982c6f4054fb43a4222a347a71461cee91bb498c"} Jul 15 15:14:50.103243 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:50.103192 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podStartSLOduration=1.103177145 podStartE2EDuration="1.103177145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:14:50.100018307 +0000 UTC m=+2460.509637580" watchObservedRunningTime="2026-07-15 15:14:50.103177145 +0000 UTC m=+2460.512796422" Jul 15 15:14:50.414941 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:50.414915 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:50.420240 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:50.420209 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:51.084558 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:51.084527 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:51.085481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:51.085462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:14:57.118951 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.118919 2567 generic.go:356] "Generic (PLEG): container finished" podID="decd5961-d849-4980-93e3-55807fe7ea37" containerID="878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b" exitCode=0 Jul 15 15:14:57.119458 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.118988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerDied","Data":"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b"} Jul 15 15:14:57.119458 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.119405 2567 scope.go:117] "RemoveContainer" containerID="878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b" Jul 15 15:14:57.120443 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.120415 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:57.120512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.120496 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:57.120559 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:57.120544 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:14:58.032986 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:58.032931 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:58.126998 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:58.126963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerStarted","Data":"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67"} Jul 15 15:14:58.299481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:58.299439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:14:58.300859 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:58.300827 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:14:58.300996 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:58.300877 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:14:59.299878 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:59.299843 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:14:59.300327 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:14:59.299898 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:00.300340 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:00.300300 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:15:00.300821 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:00.300365 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:01.300209 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:01.300175 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:15:01.300390 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:01.300237 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:02.300275 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:02.300233 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:15:02.300455 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:02.300302 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:03.299714 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:03.299674 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:15:03.299889 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:03.299730 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:04.300023 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:04.299986 2567 patch_prober.go:28] interesting pod/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 15 15:15:04.300417 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:04.300043 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 15 15:15:05.178226 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.178192 2567 generic.go:356] "Generic (PLEG): container finished" podID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerID="d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1" exitCode=0 Jul 15 15:15:05.178427 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.178266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerDied","Data":"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1"} Jul 15 15:15:05.178626 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.178611 2567 scope.go:117] "RemoveContainer" containerID="d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1" Jul 15 15:15:05.179564 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.179534 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:15:05.179669 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.179603 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:15:05.179669 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.179637 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 15:15:05.303484 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.303448 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:15:05.303876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.303837 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:15:05.304468 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:05.304446 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:15:06.084833 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:06.084799 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:15:06.184720 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:06.184684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerStarted","Data":"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4"} Jul 15 15:15:06.415096 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:06.415006 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:15:06.415931 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:06.415895 2567 patch_prober.go:28] interesting pod/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 15 15:15:06.416099 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:06.415974 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 15 15:15:07.414466 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:07.414432 2567 patch_prober.go:28] interesting pod/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 15 15:15:07.414654 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:07.414488 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 15 15:15:08.414657 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:08.414616 2567 patch_prober.go:28] interesting pod/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 15 15:15:08.415174 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:08.414687 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 15 15:15:09.415287 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:09.415245 2567 patch_prober.go:28] interesting pod/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 15 15:15:09.415704 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:09.415310 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 15 15:15:10.415126 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:10.415087 2567 patch_prober.go:28] interesting pod/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 15 15:15:10.415314 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:10.415155 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 15 15:15:11.418978 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:11.418946 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:15:11.419402 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:11.419197 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:15:11.419820 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:15:11.419800 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:18:50.299650 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:18:50.299571 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:18:50.309262 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:18:50.309241 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v25jf_f3f1225f-4a90-4e01-847d-2fe17310bada/ovn-acl-logging/0.log" Jul 15 15:21:36.261373 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:36.261338 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:21:36.261882 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:36.261551 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" containerID="cri-o://eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67" gracePeriod=30 Jul 15 15:21:41.517206 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.517178 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:21:41.556414 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556386 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556439 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556464 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556502 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556566 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556556 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wf8h\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556588 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556629 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556681 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token\") pod \"decd5961-d849-4980-93e3-55807fe7ea37\" (UID: \"decd5961-d849-4980-93e3-55807fe7ea37\") " Jul 15 15:21:41.556776 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:41.557033 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556824 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:41.557033 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.556963 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:21:41.557166 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.557084 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.557166 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.557105 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decd5961-d849-4980-93e3-55807fe7ea37-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.557166 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.557118 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.557294 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.557180 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:41.557294 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.557243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data" (OuterVolumeSpecName: "istio-data") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:41.559031 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.558999 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:41.559276 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.559256 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token" (OuterVolumeSpecName: "istio-token") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:21:41.559364 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.559282 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:21:41.559364 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.559302 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h" (OuterVolumeSpecName: "kube-api-access-8wf8h") pod "decd5961-d849-4980-93e3-55807fe7ea37" (UID: "decd5961-d849-4980-93e3-55807fe7ea37"). InnerVolumeSpecName "kube-api-access-8wf8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:21:41.658428 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658391 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.658428 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658425 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.658428 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658434 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.658428 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658442 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wf8h\" (UniqueName: \"kubernetes.io/projected/decd5961-d849-4980-93e3-55807fe7ea37-kube-api-access-8wf8h\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.658669 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658451 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decd5961-d849-4980-93e3-55807fe7ea37-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:41.658669 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:41.658461 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decd5961-d849-4980-93e3-55807fe7ea37-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:21:42.360346 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.360309 2567 generic.go:356] "Generic (PLEG): container finished" podID="decd5961-d849-4980-93e3-55807fe7ea37" containerID="eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67" exitCode=0 Jul 15 15:21:42.360569 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.360378 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" Jul 15 15:21:42.360569 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.360400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerDied","Data":"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67"} Jul 15 15:21:42.360569 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.360500 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns" event={"ID":"decd5961-d849-4980-93e3-55807fe7ea37","Type":"ContainerDied","Data":"a3451fb7103082e233cff0846233ed0ca9d1c8243444629835ff53963f6d9425"} Jul 15 15:21:42.360569 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.360532 2567 scope.go:117] "RemoveContainer" containerID="eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67" Jul 15 15:21:42.370092 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.370078 2567 scope.go:117] "RemoveContainer" containerID="878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b" Jul 15 15:21:42.380185 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.380004 2567 scope.go:117] "RemoveContainer" containerID="eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67" Jul 15 15:21:42.380267 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.380247 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:21:42.380318 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:21:42.380280 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67\": container with ID starting with eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67 not found: ID does not exist" containerID="eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67" Jul 15 15:21:42.380357 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.380314 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67"} err="failed to get container status \"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67\": rpc error: code = NotFound desc = could not find container \"eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67\": container with ID starting with eb0d62311fb63892b5babee3e068b42f1fcc65b6abbb068cbfbc4aade6881d67 not found: ID does not exist" Jul 15 15:21:42.380357 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.380333 2567 scope.go:117] "RemoveContainer" containerID="878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b" Jul 15 15:21:42.380568 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:21:42.380547 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b\": container with ID starting with 878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b not found: ID does not exist" containerID="878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b" Jul 15 15:21:42.380625 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.380578 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b"} err="failed to get container status \"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b\": rpc error: code = NotFound desc = could not find container \"878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b\": container with ID starting with 878c41634cc6216e474bb39f7bc3ec9b94912cf4ac13301e24b4e27f86af607b not found: ID does not exist" Jul 15 15:21:42.385682 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:42.385661 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0c81d7-openshift-default-7448999767-kvzns"] Jul 15 15:21:44.165004 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:21:44.164972 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="decd5961-d849-4980-93e3-55807fe7ea37" path="/var/lib/kubelet/pods/decd5961-d849-4980-93e3-55807fe7ea37/volumes" Jul 15 15:22:02.405259 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:02.405216 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:22:02.405665 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:02.405496 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" containerID="cri-o://b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4" gracePeriod=30 Jul 15 15:22:07.665258 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.665188 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:22:07.797615 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797575 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797615 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797616 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797650 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797667 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797689 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797712 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797747 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797811 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxbg2\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.797883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797837 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket\") pod \"e71d3c53-9955-4c0c-8638-305d015e8c7a\" (UID: \"e71d3c53-9955-4c0c-8638-305d015e8c7a\") " Jul 15 15:22:07.798290 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.797900 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:22:07.798290 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798012 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:22:07.798290 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798127 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:22:07.798290 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798271 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:22:07.798492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798332 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d3c53-9955-4c0c-8638-305d015e8c7a-istiod-ca-cert\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.798492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798353 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-certs\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.798492 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798366 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-workload-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.798752 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.798729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:22:07.800023 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.800000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2" (OuterVolumeSpecName: "kube-api-access-pxbg2") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "kube-api-access-pxbg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:22:07.800023 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.799994 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:22:07.800452 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.800430 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:22:07.800452 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.800441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e71d3c53-9955-4c0c-8638-305d015e8c7a" (UID: "e71d3c53-9955-4c0c-8638-305d015e8c7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:22:07.899152 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899109 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-data\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.899152 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899143 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-token\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.899152 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899156 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-podinfo\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.899411 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899170 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-istio-envoy\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.899411 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899184 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxbg2\" (UniqueName: \"kubernetes.io/projected/e71d3c53-9955-4c0c-8638-305d015e8c7a-kube-api-access-pxbg2\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:07.899411 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:07.899197 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d3c53-9955-4c0c-8638-305d015e8c7a-credential-socket\") on node \"ip-10-0-141-12.ec2.internal\" DevicePath \"\"" Jul 15 15:22:08.510512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.510413 2567 generic.go:356] "Generic (PLEG): container finished" podID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerID="b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4" exitCode=0 Jul 15 15:22:08.510512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.510462 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" Jul 15 15:22:08.510512 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.510464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerDied","Data":"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4"} Jul 15 15:22:08.510793 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.510539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq" event={"ID":"e71d3c53-9955-4c0c-8638-305d015e8c7a","Type":"ContainerDied","Data":"7eaec0b6c3433364e3d769ef982c6f4054fb43a4222a347a71461cee91bb498c"} Jul 15 15:22:08.510793 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.510567 2567 scope.go:117] "RemoveContainer" containerID="b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4" Jul 15 15:22:08.526847 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.526671 2567 scope.go:117] "RemoveContainer" containerID="d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1" Jul 15 15:22:08.530775 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.530747 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:22:08.534744 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.534709 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b2aed6-openshift-default-64c8b7798c-fdmtq"] Jul 15 15:22:08.539867 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.539841 2567 scope.go:117] "RemoveContainer" containerID="b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4" Jul 15 15:22:08.540224 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:22:08.540201 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4\": container with ID starting with b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4 not found: ID does not exist" containerID="b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4" Jul 15 15:22:08.540274 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.540237 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4"} err="failed to get container status \"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4\": rpc error: code = NotFound desc = could not find container \"b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4\": container with ID starting with b15d2916b83db81522e06a47b5d1b04108af9543c3db08cb3f2a11d885f7dfe4 not found: ID does not exist" Jul 15 15:22:08.540274 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.540257 2567 scope.go:117] "RemoveContainer" containerID="d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1" Jul 15 15:22:08.540566 ip-10-0-141-12 kubenswrapper[2567]: E0715 15:22:08.540545 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1\": container with ID starting with d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1 not found: ID does not exist" containerID="d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1" Jul 15 15:22:08.540610 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.540573 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1"} err="failed to get container status \"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1\": rpc error: code = NotFound desc = could not find container \"d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1\": container with ID starting with d6176f9ece86695841af9a478bd48f46a04981e2e9469717536f333612311df1 not found: ID does not exist" Jul 15 15:22:08.555324 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.555296 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6nccc_ab6a9a40-9a37-451b-a989-245300788082/manager/0.log" Jul 15 15:22:08.679852 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.679801 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t9r6h_7c8e7cd2-c069-4fc8-be78-bfbf10517c69/manager/0.log" Jul 15 15:22:08.799963 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.799936 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7565848f6c-77z2q_2faefc27-2adf-4719-ad54-988323933549/manager/0.log" Jul 15 15:22:08.923970 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:08.923937 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8td92_605132e6-3ba0-40a3-97a6-db746c1c171e/server/0.log" Jul 15 15:22:09.036812 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:09.036781 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b75mx_686bb40f-25d6-4b47-a59f-e307d4e306bb/manager/0.log" Jul 15 15:22:09.281377 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:09.281298 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt_65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d/copy-manifests/0.log" Jul 15 15:22:09.294097 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:09.294069 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt_65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d/manager/0.log" Jul 15 15:22:09.778430 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:09.778397 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7966b86-dldrb_e4d00c76-b9bd-4145-9d39-ce79f1f628c7/maas-api/0.log" Jul 15 15:22:10.001215 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:10.001178 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735460-pmlbm_678de9be-6c9b-456d-8109-ea1042a5503f/cleanup/0.log" Jul 15 15:22:10.164934 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:10.164899 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" path="/var/lib/kubelet/pods/e71d3c53-9955-4c0c-8638-305d015e8c7a/volumes" Jul 15 15:22:10.227478 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:10.227447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-8a5fa064-f6vzc_d33a69da-8f87-425b-b2ea-c665d3377be8/revoke-keys/0.log" Jul 15 15:22:11.009972 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:11.009943 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-fc2c3cfa-v8xvd_ecc0ccfc-98f0-4e25-9abb-5011b7492fe3/revoke-keys/0.log" Jul 15 15:22:11.569740 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:11.569711 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qklxh_59b80307-6fed-4352-83ba-4c6c959331f7/postgres/0.log" Jul 15 15:22:12.331348 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.331306 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/util/0.log" Jul 15 15:22:12.343455 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.343424 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/pull/0.log" Jul 15 15:22:12.351047 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.351028 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/extract/0.log" Jul 15 15:22:12.470377 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.470351 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/util/0.log" Jul 15 15:22:12.477136 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.477114 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/pull/0.log" Jul 15 15:22:12.483883 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.483865 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/extract/0.log" Jul 15 15:22:12.595642 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.595565 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/util/0.log" Jul 15 15:22:12.602911 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.602890 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/pull/0.log" Jul 15 15:22:12.609728 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.609707 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/extract/0.log" Jul 15 15:22:12.734165 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.734138 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/util/0.log" Jul 15 15:22:12.741139 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.741119 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/pull/0.log" Jul 15 15:22:12.748265 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:12.748246 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/extract/0.log" Jul 15 15:22:13.478726 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:13.478695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pw46f_8535db98-63f7-42ff-9774-4d864939faf7/manager/0.log" Jul 15 15:22:14.194596 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:14.194570 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dg7kv_445e192e-a9b9-4198-82c9-0e5ea71d9dc7/discovery/0.log" Jul 15 15:22:14.425184 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:14.425155 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d556468c-t8vkk_c26196e2-66c6-474a-9eb9-17268e4dba2f/kube-auth-proxy/0.log" Jul 15 15:22:14.662362 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:14.662333 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5qw62_c57d03ab-fd6d-46a1-986b-c632b83c4abb/payload-pre-processing/0.log" Jul 15 15:22:14.775201 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:14.775172 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-nfb5w_311fa312-9e09-40df-a78e-6a8a3cd1227d/payload-processing/0.log" Jul 15 15:22:15.353382 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:15.353353 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-z2cq6_41d570ad-ae52-4704-8b84-ec78d5e6f866/main/0.log" Jul 15 15:22:24.015198 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:24.015165 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-864x5_2a835c6a-ccd5-40db-b075-a1437d022fc1/global-pull-secret-syncer/0.log" Jul 15 15:22:24.163071 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:24.163025 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p67nb_ae03a625-f5e7-4b8f-9b99-4c704f4c9366/konnectivity-agent/0.log" Jul 15 15:22:24.228338 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:24.228312 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-12.ec2.internal_17256dde85068a9f5597774cee525f87/haproxy/0.log" Jul 15 15:22:28.703312 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.703273 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/extract/0.log" Jul 15 15:22:28.730953 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.730925 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/util/0.log" Jul 15 15:22:28.753876 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.753846 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jrdkn_aac858ac-67c1-4bc2-bae5-cf5515a5ac73/pull/0.log" Jul 15 15:22:28.792350 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.792315 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/extract/0.log" Jul 15 15:22:28.819026 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.819000 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/util/0.log" Jul 15 15:22:28.842435 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.842410 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f4sbt_61340d45-9b18-4baf-8ded-667e3bf32e08/pull/0.log" Jul 15 15:22:28.872258 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.872230 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/extract/0.log" Jul 15 15:22:28.896660 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.896629 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/util/0.log" Jul 15 15:22:28.921178 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.921147 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xwbv5_834e72c5-f717-464f-ab98-94b84775339e/pull/0.log" Jul 15 15:22:28.969109 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.969013 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/extract/0.log" Jul 15 15:22:28.994499 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:28.994468 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/util/0.log" Jul 15 15:22:29.018994 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:29.018962 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x4pwl_7fe2e190-a4f2-4158-89d6-0c85c11f3d41/pull/0.log" Jul 15 15:22:29.632429 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:29.632396 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pw46f_8535db98-63f7-42ff-9774-4d864939faf7/manager/0.log" Jul 15 15:22:31.719930 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:31.719901 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5kd7p_fc89f221-3793-40f7-83dc-bd0adf685698/kube-state-metrics/0.log" Jul 15 15:22:31.740301 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:31.740274 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5kd7p_fc89f221-3793-40f7-83dc-bd0adf685698/kube-rbac-proxy-main/0.log" Jul 15 15:22:31.766481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:31.766453 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5kd7p_fc89f221-3793-40f7-83dc-bd0adf685698/kube-rbac-proxy-self/0.log" Jul 15 15:22:31.799546 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:31.799511 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f67b9cd4-xnmnp_778fd302-84b5-46be-8c74-9a249efbfd80/metrics-server/0.log" Jul 15 15:22:31.828507 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:31.828476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-csnwg_997666de-8abc-471c-a073-02bbf8412f27/monitoring-plugin/0.log" Jul 15 15:22:32.062887 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:32.062859 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6zrw_455d8a67-95db-4f62-8224-00ef2e4208ad/node-exporter/0.log" Jul 15 15:22:32.088489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:32.088465 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6zrw_455d8a67-95db-4f62-8224-00ef2e4208ad/kube-rbac-proxy/0.log" Jul 15 15:22:32.110044 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:32.110019 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6zrw_455d8a67-95db-4f62-8224-00ef2e4208ad/init-textfile/0.log" Jul 15 15:22:32.480151 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:32.480044 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-bmmf9_a3ae71fb-8de0-45c4-a2e9-e3ab9d7acc35/prometheus-operator-admission-webhook/0.log" Jul 15 15:22:35.067525 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.067495 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fd5f58986-x7d6p_809d3aa7-81e9-4ea3-bc04-a7b32fbb7252/console/0.log" Jul 15 15:22:35.563825 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.563781 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk"] Jul 15 15:22:35.564315 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564295 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.564315 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564317 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564332 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564340 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564349 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564357 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564443 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71d3c53-9955-4c0c-8638-305d015e8c7a" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564455 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.564489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.564464 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="decd5961-d849-4980-93e3-55807fe7ea37" containerName="istio-proxy" Jul 15 15:22:35.566637 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.566616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.568611 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.568590 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bqlw8\"/\"kube-root-ca.crt\"" Jul 15 15:22:35.568715 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.568617 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bqlw8\"/\"openshift-service-ca.crt\"" Jul 15 15:22:35.568859 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.568846 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bqlw8\"/\"default-dockercfg-dd5lq\"" Jul 15 15:22:35.574255 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.573501 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk"] Jul 15 15:22:35.638805 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.638770 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-podres\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.639022 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.638813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmvm\" (UniqueName: \"kubernetes.io/projected/450d80a0-ae81-458d-ad2d-9eff80e74fe2-kube-api-access-ntmvm\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.639022 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.638838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-lib-modules\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.639022 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.638885 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-proc\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.639022 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.638963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-sys\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740094 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-podres\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmvm\" (UniqueName: \"kubernetes.io/projected/450d80a0-ae81-458d-ad2d-9eff80e74fe2-kube-api-access-ntmvm\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-lib-modules\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-proc\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-sys\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740223 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-proc\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-sys\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740219 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-podres\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.740289 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.740281 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450d80a0-ae81-458d-ad2d-9eff80e74fe2-lib-modules\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.748681 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.748654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmvm\" (UniqueName: \"kubernetes.io/projected/450d80a0-ae81-458d-ad2d-9eff80e74fe2-kube-api-access-ntmvm\") pod \"perf-node-gather-daemonset-4cvtk\" (UID: \"450d80a0-ae81-458d-ad2d-9eff80e74fe2\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:35.878279 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:35.878191 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:36.018348 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.018321 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk"] Jul 15 15:22:36.021111 ip-10-0-141-12 kubenswrapper[2567]: W0715 15:22:36.021044 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod450d80a0_ae81_458d_ad2d_9eff80e74fe2.slice/crio-5c0f20a22da614b71fd5fbeeeaf45c65bbca7af3ddbc017ee23433fca8f7d28c WatchSource:0}: Error finding container 5c0f20a22da614b71fd5fbeeeaf45c65bbca7af3ddbc017ee23433fca8f7d28c: Status 404 returned error can't find the container with id 5c0f20a22da614b71fd5fbeeeaf45c65bbca7af3ddbc017ee23433fca8f7d28c Jul 15 15:22:36.665128 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.665084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" event={"ID":"450d80a0-ae81-458d-ad2d-9eff80e74fe2","Type":"ContainerStarted","Data":"c631242ca834ad758d5cbc23e09281bb4b08bc02afcb50ae839b81a025d11ce3"} Jul 15 15:22:36.665535 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.665143 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" event={"ID":"450d80a0-ae81-458d-ad2d-9eff80e74fe2","Type":"ContainerStarted","Data":"5c0f20a22da614b71fd5fbeeeaf45c65bbca7af3ddbc017ee23433fca8f7d28c"} Jul 15 15:22:36.665535 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.665170 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:36.683814 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.683758 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" podStartSLOduration=1.683742708 podStartE2EDuration="1.683742708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 15:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:22:36.680868363 +0000 UTC m=+2927.090487641" watchObservedRunningTime="2026-07-15 15:22:36.683742708 +0000 UTC m=+2927.093361987" Jul 15 15:22:36.924660 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.924561 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bd88j_926b471b-8345-4e7a-828f-931d85a39e04/dns/0.log" Jul 15 15:22:36.946494 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:36.946466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bd88j_926b471b-8345-4e7a-828f-931d85a39e04/kube-rbac-proxy/0.log" Jul 15 15:22:37.046684 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:37.046653 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77j4x_716f201f-0bc0-4fb7-92a4-124c180171bd/dns-node-resolver/0.log" Jul 15 15:22:37.630857 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:37.630822 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bfbb8fbb9-pwttq_bc93a353-0e25-49a9-b35c-0800dcc8aeff/registry/0.log" Jul 15 15:22:37.676575 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:37.676549 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nzwnl_d13942e3-ce89-4455-9a17-7681469756ce/node-ca/0.log" Jul 15 15:22:38.840535 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:38.840501 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dg7kv_445e192e-a9b9-4198-82c9-0e5ea71d9dc7/discovery/0.log" Jul 15 15:22:38.891398 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:38.891362 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d556468c-t8vkk_c26196e2-66c6-474a-9eb9-17268e4dba2f/kube-auth-proxy/0.log" Jul 15 15:22:38.979125 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:38.979086 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5qw62_c57d03ab-fd6d-46a1-986b-c632b83c4abb/payload-pre-processing/0.log" Jul 15 15:22:39.009535 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:39.009505 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-nfb5w_311fa312-9e09-40df-a78e-6a8a3cd1227d/payload-processing/0.log" Jul 15 15:22:39.798359 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:39.798329 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m5dj6_3d249ec3-554d-48f7-b365-e22dc3f2d482/serve-healthcheck-canary/0.log" Jul 15 15:22:40.362166 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:40.362129 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bjkb_0c338dfc-d036-4d2a-96cb-9e35564de89e/kube-rbac-proxy/0.log" Jul 15 15:22:40.386648 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:40.386620 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bjkb_0c338dfc-d036-4d2a-96cb-9e35564de89e/exporter/0.log" Jul 15 15:22:40.408489 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:40.408466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bjkb_0c338dfc-d036-4d2a-96cb-9e35564de89e/extractor/0.log" Jul 15 15:22:42.681940 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:42.681910 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-4cvtk" Jul 15 15:22:42.705825 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:42.705796 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6nccc_ab6a9a40-9a37-451b-a989-245300788082/manager/0.log" Jul 15 15:22:42.862221 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:42.862187 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t9r6h_7c8e7cd2-c069-4fc8-be78-bfbf10517c69/manager/0.log" Jul 15 15:22:42.962017 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:42.961936 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7565848f6c-77z2q_2faefc27-2adf-4719-ad54-988323933549/manager/0.log" Jul 15 15:22:42.982798 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:42.982775 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8td92_605132e6-3ba0-40a3-97a6-db746c1c171e/server/0.log" Jul 15 15:22:43.052606 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:43.052572 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b75mx_686bb40f-25d6-4b47-a59f-e307d4e306bb/manager/0.log" Jul 15 15:22:43.181277 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:43.181234 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt_65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d/manager/0.log" Jul 15 15:22:43.198263 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:43.198241 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-mlkxt_65424aea-3b5b-490c-ac9f-f2a1a6cb5e4d/copy-manifests/0.log" Jul 15 15:22:44.475823 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:44.475790 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dg7kv_445e192e-a9b9-4198-82c9-0e5ea71d9dc7/discovery/0.log" Jul 15 15:22:44.509452 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:44.509424 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d556468c-t8vkk_c26196e2-66c6-474a-9eb9-17268e4dba2f/kube-auth-proxy/0.log" Jul 15 15:22:44.580481 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:44.580449 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5qw62_c57d03ab-fd6d-46a1-986b-c632b83c4abb/payload-pre-processing/0.log" Jul 15 15:22:44.600040 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:44.599998 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-nfb5w_311fa312-9e09-40df-a78e-6a8a3cd1227d/payload-processing/0.log" Jul 15 15:22:44.644925 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:44.644890 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wlvd6_8b58961a-dd47-4cf5-8956-a8da8b3bbe26/openshift-lws-operator/0.log" Jul 15 15:22:46.081499 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:46.081468 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dg7kv_445e192e-a9b9-4198-82c9-0e5ea71d9dc7/discovery/0.log" Jul 15 15:22:46.111155 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:46.111128 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d556468c-t8vkk_c26196e2-66c6-474a-9eb9-17268e4dba2f/kube-auth-proxy/0.log" Jul 15 15:22:46.182457 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:46.182426 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5qw62_c57d03ab-fd6d-46a1-986b-c632b83c4abb/payload-pre-processing/0.log" Jul 15 15:22:46.202700 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:46.202671 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-nfb5w_311fa312-9e09-40df-a78e-6a8a3cd1227d/payload-processing/0.log" Jul 15 15:22:51.487787 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.487750 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-69q2k_88524ec0-8260-48f6-adbe-a9fa04257b25/kube-multus/0.log" Jul 15 15:22:51.845455 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.845426 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/kube-multus-additional-cni-plugins/0.log" Jul 15 15:22:51.895105 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.895078 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/egress-router-binary-copy/0.log" Jul 15 15:22:51.928580 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.928554 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/cni-plugins/0.log" Jul 15 15:22:51.949960 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.949937 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/bond-cni-plugin/0.log" Jul 15 15:22:51.973230 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:51.973212 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/routeoverride-cni/0.log" Jul 15 15:22:52.000628 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:52.000607 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/whereabouts-cni-bincopy/0.log" Jul 15 15:22:52.022795 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:52.022772 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t9474_62e20b9a-ec6b-4119-8892-4666e75413fb/whereabouts-cni/0.log" Jul 15 15:22:52.099992 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:52.099904 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9z5t2_a736a3ad-c62f-45ec-974e-2902d5c4ee62/network-metrics-daemon/0.log" Jul 15 15:22:52.118786 ip-10-0-141-12 kubenswrapper[2567]: I0715 15:22:52.118763 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9z5t2_a736a3ad-c62f-45ec-974e-2902d5c4ee62/kube-rbac-proxy/0.log"