Jul 09 12:00:27.942466 ip-10-0-133-7 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 12:00:27.942478 ip-10-0-133-7 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 12:00:27.942488 ip-10-0-133-7 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 12:00:27.942782 ip-10-0-133-7 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 12:00:38.158347 ip-10-0-133-7 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 12:00:38.158370 ip-10-0-133-7 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5be362dbf256492cb5d32ac66c5c69bd -- Jul 09 12:02:43.276208 ip-10-0-133-7 systemd[1]: Starting Kubernetes Kubelet... Jul 09 12:02:43.716465 ip-10-0-133-7 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 12:02:43.716465 ip-10-0-133-7 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 12:02:43.716465 ip-10-0-133-7 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 12:02:43.716465 ip-10-0-133-7 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 12:02:43.716465 ip-10-0-133-7 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 12:02:43.717642 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.717392 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723937 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723957 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723962 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723965 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723968 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723971 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723974 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723976 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723979 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.723970 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723982 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723985 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723988 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723991 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723993 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723996 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.723998 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724001 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724003 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724006 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724008 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724011 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724013 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724015 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724018 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724020 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724023 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724026 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724028 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724031 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.724306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724033 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724035 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724038 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724047 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724050 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724052 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724055 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724057 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724059 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724062 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724064 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724067 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724069 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724072 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724076 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724078 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724081 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724083 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724086 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724090 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.724788 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724094 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724097 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724100 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724103 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724106 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724108 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724111 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724113 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724116 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724118 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724121 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724123 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724127 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724130 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724133 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724136 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724138 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724141 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724143 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724146 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.725291 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724148 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724152 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724154 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724156 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724159 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724161 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724164 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724167 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724169 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724172 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724174 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724177 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724179 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724182 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724185 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724188 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724191 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724193 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724195 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724198 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.725781 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724201 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724203 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724206 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724208 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724211 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724213 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724216 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724218 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724221 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724224 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724982 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724990 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724993 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.724997 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725000 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725003 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725006 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725009 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725013 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.726329 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725016 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725019 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725022 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725025 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725028 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725031 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725034 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725037 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725040 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725043 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725045 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725048 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725051 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725053 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725056 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725058 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725060 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725063 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725065 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725068 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.726868 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725070 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725073 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725075 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725078 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725081 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725084 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725087 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725089 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725092 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725094 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725097 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725099 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725102 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725105 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725107 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725110 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725113 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725115 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725117 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.727394 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725121 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725123 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725126 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725129 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725131 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725134 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725137 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725140 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725143 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725145 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725148 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725150 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725152 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725155 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725157 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725161 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725165 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725167 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725170 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725172 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.727869 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725174 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725177 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725179 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725182 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725184 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725187 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725190 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725192 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725195 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725197 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725200 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725202 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725204 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725207 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725209 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725214 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725216 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725219 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725221 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725224 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.728385 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725226 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725230 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725233 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725236 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725238 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725241 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725243 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725246 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725250 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725253 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725256 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725651 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725657 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725660 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725663 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725667 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725669 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725672 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.728885 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725675 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725678 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725680 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725683 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725686 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725688 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725690 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725693 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725695 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725698 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725701 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725703 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725706 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725708 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725711 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725713 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725715 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725718 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725720 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725723 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.729349 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725726 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725730 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725733 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725735 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725738 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725740 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725743 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725746 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725748 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725751 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725754 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725756 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725759 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725761 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725764 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725767 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725769 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725773 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725776 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.729839 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725779 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725782 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725785 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725788 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725790 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725793 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725796 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725798 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725801 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725803 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725806 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725808 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725810 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725813 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725815 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725818 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725820 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725823 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725827 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.730335 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725829 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725832 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725834 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725837 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725840 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725842 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725845 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725847 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725850 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725852 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725854 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725857 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725859 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725862 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725864 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725866 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725869 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725872 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725874 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725877 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.730802 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725879 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725881 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725883 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725886 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725888 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725891 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725907 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725910 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725914 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725917 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725920 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725922 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725924 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.725927 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.725996 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726007 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726016 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726020 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726024 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726027 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726031 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 12:02:43.731355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726035 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726039 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726041 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726045 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726049 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726052 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726055 2567 flags.go:64] FLAG: --cgroup-root="" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726057 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726060 2567 flags.go:64] FLAG: --client-ca-file="" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726064 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726066 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726071 2567 flags.go:64] FLAG: --cluster-domain="" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726074 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726077 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726080 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726083 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726087 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726089 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726092 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726095 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726098 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726101 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726104 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726107 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 12:02:43.731866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726110 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726114 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726119 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726122 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726125 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726128 2567 flags.go:64] FLAG: --enable-server="true" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726131 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726135 2567 flags.go:64] FLAG: --event-burst="100" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726138 2567 flags.go:64] FLAG: --event-qps="50" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726141 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726144 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726147 2567 flags.go:64] FLAG: --eviction-hard="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726151 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726153 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726156 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726159 2567 flags.go:64] FLAG: --eviction-soft="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726162 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726164 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726167 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726170 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726173 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726176 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726179 2567 flags.go:64] FLAG: --feature-gates="" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726182 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726185 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 12:02:43.732459 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726188 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726190 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726194 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726197 2567 flags.go:64] FLAG: --help="false" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726199 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-133-7.ec2.internal" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726202 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726205 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726208 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726211 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726215 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726218 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726221 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726224 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726226 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726229 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726232 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726235 2567 flags.go:64] FLAG: --kube-reserved="" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726238 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726240 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726243 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726246 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726249 2567 flags.go:64] FLAG: --lock-file="" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726251 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726254 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 12:02:43.733124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726257 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726263 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726267 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726269 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726272 2567 flags.go:64] FLAG: --logging-format="text" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726275 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726278 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726280 2567 flags.go:64] FLAG: --manifest-url="" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726283 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726291 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726294 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726298 2567 flags.go:64] FLAG: --max-pods="110" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726301 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726304 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726306 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726309 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726312 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726316 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726319 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726326 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726329 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726332 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726336 2567 flags.go:64] FLAG: --pod-cidr="" Jul 09 12:02:43.733738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726339 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726344 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726347 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726350 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726353 2567 flags.go:64] FLAG: --port="10250" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726356 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726359 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-019e4162cc63b4d6f" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726362 2567 flags.go:64] FLAG: --qos-reserved="" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726365 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726368 2567 flags.go:64] FLAG: --register-node="true" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726371 2567 flags.go:64] FLAG: --register-with-taints="" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726374 2567 flags.go:64] FLAG: --registry-burst="10" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726378 2567 flags.go:64] FLAG: --registry-qps="5" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726380 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726383 2567 flags.go:64] FLAG: --reserved-memory="" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726387 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726390 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726393 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726396 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726399 2567 flags.go:64] FLAG: --runonce="false" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726402 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726404 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726407 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726410 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726413 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726416 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 12:02:43.734328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726418 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726423 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726425 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726428 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726431 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726434 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726437 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726439 2567 flags.go:64] FLAG: --system-cgroups="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726442 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726447 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726449 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726452 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726456 2567 flags.go:64] FLAG: --tls-min-version="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726459 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726463 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726466 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726469 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726471 2567 flags.go:64] FLAG: --v="2" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726475 2567 flags.go:64] FLAG: --version="false" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726480 2567 flags.go:64] FLAG: --vmodule="" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726484 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.726487 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726601 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726604 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.735088 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726607 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726610 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726612 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726615 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726617 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726620 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726623 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726625 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726628 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726631 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726634 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726636 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726639 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726641 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726644 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726646 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726649 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726652 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726656 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.735660 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726659 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726662 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726665 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726669 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726671 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726674 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726676 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726679 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726681 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726684 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726687 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726689 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726692 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726694 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726698 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726701 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726704 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726706 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726709 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726711 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.736141 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726714 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726716 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726720 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726723 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726725 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726727 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726730 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726732 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726735 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726737 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726740 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726742 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726745 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726747 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726750 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726753 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726756 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726759 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726762 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726764 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.736623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726767 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726769 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726773 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726776 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726780 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726783 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726785 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726787 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726790 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726793 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726795 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726797 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726800 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726802 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726806 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726808 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726811 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726813 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726816 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.737134 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726818 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726821 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726824 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726826 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726829 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726831 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726834 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726836 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726840 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726842 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726844 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726847 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726849 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726851 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726854 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726856 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726859 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726862 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.737597 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.726864 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.727444 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.734472 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.734486 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.734505 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.734511 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734568 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734573 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734576 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734579 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734581 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734585 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734587 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734590 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734593 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.738076 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734596 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734598 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734601 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734604 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734606 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734609 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734612 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734614 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734617 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734619 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734621 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734624 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734626 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734629 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734631 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734633 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734636 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734638 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734640 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734643 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.738468 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734647 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734650 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734652 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734655 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734658 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734662 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734666 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734668 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734671 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734674 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734676 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734679 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734681 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734684 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734686 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734689 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734692 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734695 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734697 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734700 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.738980 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734702 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734704 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734707 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734709 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734712 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734714 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734716 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734719 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734724 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734727 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734730 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734733 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734736 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734739 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734742 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734745 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734747 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734750 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734752 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.739497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734754 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734757 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734759 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734762 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734765 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734767 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734769 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734772 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734774 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734777 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734781 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734784 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734787 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734789 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734792 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734795 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734797 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734800 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734802 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.740056 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734805 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734807 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734810 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734812 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734814 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734818 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734821 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734824 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734830 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734833 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734836 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734838 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.734843 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734977 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734983 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 12:02:43.740527 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734986 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734989 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734992 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734994 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734997 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.734999 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735002 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735004 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735006 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735009 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735011 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735013 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735016 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735018 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735020 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735023 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735025 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735028 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735030 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735032 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 12:02:43.740922 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735035 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735037 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735039 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735043 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735046 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735048 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735051 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735054 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735056 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735059 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735062 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735066 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735069 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735072 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735076 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735079 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735081 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735083 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 12:02:43.741405 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735086 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735089 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735091 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735094 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735097 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735099 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735102 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735105 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735107 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735110 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735112 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735114 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735117 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735119 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735122 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735124 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735127 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735130 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735133 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735135 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 12:02:43.741921 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735138 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735141 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735143 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735146 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735148 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735151 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735153 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735155 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735158 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735160 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735162 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735165 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735167 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735169 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735172 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735174 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735177 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735179 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735181 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735184 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 12:02:43.742417 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735187 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735189 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735191 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735193 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735196 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735198 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735200 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735203 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735205 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735209 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735213 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735215 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735218 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735220 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735223 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735225 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735228 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735230 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 12:02:43.742990 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:43.735233 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.735237 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.735340 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.737485 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.738999 2567 server.go:1019] "Starting client certificate rotation" Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.739098 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.739126 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 12:02:43.743497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.741778 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 12:02:43.748743 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.748719 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 12:02:43.757210 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.757193 2567 log.go:25] "Validated CRI v1 runtime API" Jul 09 12:02:43.759694 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.759681 2567 log.go:25] "Validated CRI v1 image API" Jul 09 12:02:43.760549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.760535 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 12:02:43.765111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.765093 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9c15c3c5-6a83-4f99-8429-1df45e38cf1d:/dev/nvme0n1p4 bb227c7a-c187-49af-a88b-6a26223dd02e:/dev/nvme0n1p3] Jul 09 12:02:43.765181 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.765111 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 12:02:43.770454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.770343 2567 manager.go:216] Machine: {Timestamp:2026-07-09 12:02:43.76874149 +0000 UTC m=+0.368497272 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099478 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2772cea09abfed8ebe60e3032d58f5 SystemUUID:ec2772ce-a09a-bfed-8ebe-60e3032d58f5 BootID:5be362db-f256-492c-b5d3-2ac66c5c69bd Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:cb:18:c8:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:cb:18:c8:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:3a:87:5d:8b:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 12:02:43.770454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.770451 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 12:02:43.770561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.770531 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 12:02:43.774352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.774324 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 12:02:43.774487 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.774355 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 12:02:43.775906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.775880 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 12:02:43.775962 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.775911 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 12:02:43.775962 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.775921 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 12:02:43.776652 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.776628 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 12:02:43.776721 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.776661 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 12:02:43.778337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.778325 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 12:02:43.781740 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781729 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 09 12:02:43.781853 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781844 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 12:02:43.781883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781874 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 09 12:02:43.781927 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781884 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 12:02:43.781927 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781916 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 12:02:43.781927 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781927 2567 kubelet.go:398] "Adding apiserver pod source" Jul 09 12:02:43.782035 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.781937 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 12:02:43.782271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.782260 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 12:02:43.782337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.782277 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 12:02:43.785091 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.785063 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 12:02:43.789106 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.789088 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 12:02:43.789182 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.789121 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 12:02:43.790403 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790386 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 12:02:43.790453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790410 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 12:02:43.790453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790420 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 12:02:43.790453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790429 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 12:02:43.790453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790442 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 12:02:43.790453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790450 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790461 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790469 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790479 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790488 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790509 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 12:02:43.790595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.790517 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 12:02:43.791396 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.791385 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 12:02:43.791436 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.791398 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 12:02:43.792788 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.792759 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 12:02:43.792863 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.792818 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 12:02:43.794645 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.794632 2567 server.go:1268] "Started kubelet" Jul 09 12:02:43.794740 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.794715 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 12:02:43.794778 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.794762 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 12:02:43.794875 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.794853 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 12:02:43.795502 ip-10-0-133-7 systemd[1]: Started Kubernetes Kubelet. Jul 09 12:02:43.796109 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.796059 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 12:02:43.797002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.796986 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 09 12:02:43.801663 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.801643 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 12:02:43.802279 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.802260 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kh69m" Jul 09 12:02:43.802594 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.802581 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 12:02:43.802653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.802593 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 12:02:43.802701 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.802661 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:43.803292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803197 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 12:02:43.803476 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803455 2567 factory.go:55] Registering systemd factory Jul 09 12:02:43.803476 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803477 2567 factory.go:223] Registration of the systemd container factory successfully Jul 09 12:02:43.803476 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803486 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-7.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 12:02:43.803476 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803509 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803522 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803594 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803603 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803667 2567 factory.go:153] Registering CRI-O factory Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803676 2567 factory.go:223] Registration of the crio container factory successfully Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803725 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 12:02:43.803745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803747 2567 factory.go:103] Registering Raw factory Jul 09 12:02:43.803945 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.803760 2567 manager.go:1196] Started watching for new ooms in manager Jul 09 12:02:43.804266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.804250 2567 manager.go:349] Starting recovery of all containers Jul 09 12:02:43.806674 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.803367 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-7.ec2.internal.18c09d71a0dfd7ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-7.ec2.internal,UID:ip-10-0-133-7.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-7.ec2.internal,},FirstTimestamp:2026-07-09 12:02:43.794605995 +0000 UTC m=+0.394361758,LastTimestamp:2026-07-09 12:02:43.794605995 +0000 UTC m=+0.394361758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-7.ec2.internal,}" Jul 09 12:02:43.808171 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.808144 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 12:02:43.808258 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.808237 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 12:02:43.809376 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.809355 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kh69m" Jul 09 12:02:43.817037 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.817021 2567 manager.go:354] Recovery completed Jul 09 12:02:43.821255 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.821239 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:43.824697 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.824682 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:43.824745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.824710 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:43.824745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.824720 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:43.825297 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.825285 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 12:02:43.825361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.825297 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 12:02:43.825361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.825312 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 09 12:02:43.825361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.825287 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:43.826486 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826471 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:43.826565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826500 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:43.826565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826515 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:43.826565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826525 2567 policy_none.go:49] "None policy: Start" Jul 09 12:02:43.826565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826538 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 12:02:43.826565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.826547 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 12:02:43.827283 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.827219 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-7.ec2.internal.18c09d71a2ab01cf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-7.ec2.internal,UID:ip-10-0-133-7.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-7.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-7.ec2.internal,},FirstTimestamp:2026-07-09 12:02:43.824697807 +0000 UTC m=+0.424453570,LastTimestamp:2026-07-09 12:02:43.824697807 +0000 UTC m=+0.424453570,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-7.ec2.internal,}" Jul 09 12:02:43.827572 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.827562 2567 policy_none.go:47] "Start" Jul 09 12:02:43.862533 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.862514 2567 manager.go:337] "Starting Device Plugin manager" Jul 09 12:02:43.862633 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.862555 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 12:02:43.862633 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.862568 2567 server.go:87] "Starting device plugin registration server" Jul 09 12:02:43.862858 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.862844 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 12:02:43.862909 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.862864 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 12:02:43.863023 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.863007 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 12:02:43.863105 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.863093 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 12:02:43.863148 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.863105 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 12:02:43.863608 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.863590 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 12:02:43.863681 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.863644 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:43.963420 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.963384 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:43.964857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.964464 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:43.964857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.964491 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:43.964857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.964503 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:43.964857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.964530 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-7.ec2.internal" Jul 09 12:02:43.972444 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:43.972406 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-7.ec2.internal" Jul 09 12:02:43.972444 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.972430 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-7.ec2.internal\": node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:43.985378 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:43.985355 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:44.002491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.002450 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 12:02:44.003811 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.003786 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 12:02:44.003921 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.003832 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 12:02:44.003921 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.003855 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 12:02:44.003921 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.003893 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 12:02:44.006273 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.006255 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 12:02:44.085756 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.085706 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:44.103978 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.103952 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal"] Jul 09 12:02:44.104075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.104027 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:44.105143 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.105124 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:44.105250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.105161 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:44.105250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.105175 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:44.106627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.106611 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:44.106754 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.106739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.106802 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.106769 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:44.107384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107368 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:44.107454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107395 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:44.107454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107405 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:44.107454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107377 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:44.107549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107461 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:44.107549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.107470 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:44.108758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.108741 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.108951 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.108765 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 12:02:44.109452 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.109440 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientMemory" Jul 09 12:02:44.109512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.109463 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 12:02:44.109512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.109473 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeHasSufficientPID" Jul 09 12:02:44.135628 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.135591 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.ec2.internal\" not found" node="ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.140027 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.140011 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.ec2.internal\" not found" node="ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.185990 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.185968 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:44.204382 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.204352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.286820 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.286763 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:44.305142 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.305123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.305213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.305149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.305213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.305187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bff7cbf82451b4333adfc3db2c0112-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.ec2.internal\" (UID: \"e2bff7cbf82451b4333adfc3db2c0112\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.305274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.305256 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.387338 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:44.387299 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:44.405683 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.405656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.405751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.405689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bff7cbf82451b4333adfc3db2c0112-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.ec2.internal\" (UID: \"e2bff7cbf82451b4333adfc3db2c0112\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.405751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.405717 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bff7cbf82451b4333adfc3db2c0112-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.ec2.internal\" (UID: \"e2bff7cbf82451b4333adfc3db2c0112\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.405751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.405736 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a0e5a37365a0cf71513576d558545d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal\" (UID: \"89a0e5a37365a0cf71513576d558545d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:44.438820 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:44.438791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:46.281326 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.280691 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:46.325078 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.325044 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.425745 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.425654 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.526259 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.526226 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.575702 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.575675 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 12:02:46.576428 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.575850 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 12:02:46.627229 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.627193 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.651395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.651359 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 11:57:45 +0000 UTC" deadline="2028-03-14 20:01:39.063429313 +0000 UTC" Jul 09 12:02:46.651395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.651391 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14743h58m52.412041744s" Jul 09 12:02:46.727864 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.727785 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.767405 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.767377 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 12:02:46.828871 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.828831 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.850259 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:46.850220 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bff7cbf82451b4333adfc3db2c0112.slice/crio-2595d79eea1e89bb8bfed6520822bd8de9d3368a7de5f8a6bf92471152f1c8d9 WatchSource:0}: Error finding container 2595d79eea1e89bb8bfed6520822bd8de9d3368a7de5f8a6bf92471152f1c8d9: Status 404 returned error can't find the container with id 2595d79eea1e89bb8bfed6520822bd8de9d3368a7de5f8a6bf92471152f1c8d9 Jul 09 12:02:46.850709 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:46.850694 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a0e5a37365a0cf71513576d558545d.slice/crio-807a481840bd296aa415f60b45e78dcc75a7442537fe1aabf9024390c415a4ab WatchSource:0}: Error finding container 807a481840bd296aa415f60b45e78dcc75a7442537fe1aabf9024390c415a4ab: Status 404 returned error can't find the container with id 807a481840bd296aa415f60b45e78dcc75a7442537fe1aabf9024390c415a4ab Jul 09 12:02:46.856300 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.856275 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:02:46.929919 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:46.929884 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:46.949414 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:46.949388 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 12:02:47.030369 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.030276 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-7.ec2.internal\" not found" Jul 09 12:02:47.085291 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.085274 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 12:02:47.140438 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.140409 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" Jul 09 12:02:47.159383 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.159363 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 12:02:47.160446 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.160430 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" Jul 09 12:02:47.170277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.170261 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 12:02:47.621059 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.621032 2567 apiserver.go:52] "Watching apiserver" Jul 09 12:02:47.627967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.627941 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 12:02:47.628463 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.628440 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s","openshift-dns/node-resolver-bjdxm","openshift-multus/multus-additional-cni-plugins-6zv7s","openshift-multus/multus-vs42f","openshift-multus/network-metrics-daemon-jwdfr","openshift-network-diagnostics/network-check-target-g4wkb","kube-system/global-pull-secret-syncer-5cb4v","openshift-cluster-node-tuning-operator/tuned-fmr9l","openshift-image-registry/node-ca-88pcx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal","openshift-network-operator/iptables-alerter-4mxht","openshift-ovn-kubernetes/ovnkube-node-qt4h5","kube-system/konnectivity-agent-q9bb6"] Jul 09 12:02:47.630179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.630159 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:47.630301 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.630252 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:47.631676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.631660 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.632940 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.632913 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634531 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634558 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlzd8\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634530 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634605 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634538 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634660 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 12:02:47.634716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634650 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 12:02:47.635103 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.634958 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.635384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.635255 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.635384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.635316 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lrwnt\"" Jul 09 12:02:47.635530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.635513 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.636371 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.636356 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.637694 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.637675 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.637785 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.637764 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:47.637847 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.637772 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.638077 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638063 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 12:02:47.638334 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638321 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 12:02:47.638403 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638321 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 12:02:47.638465 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638452 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.638641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638610 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 12:02:47.638943 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.638906 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8gdkq\"" Jul 09 12:02:47.639421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.639245 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gv5fl\"" Jul 09 12:02:47.639421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.639267 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.641591 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.641544 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 12:02:47.641823 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.641797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.643131 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.643107 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.643131 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.643112 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.643326 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.643307 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qkmsd\"" Jul 09 12:02:47.644831 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.644790 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.644987 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.644968 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.645412 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.645390 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 12:02:47.645587 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.645566 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.645657 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.645566 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cnscz\"" Jul 09 12:02:47.648331 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.648315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.648448 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.648429 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.648561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.648433 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmkmw\"" Jul 09 12:02:47.648561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.648479 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.648668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.648372 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 12:02:47.649847 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.649832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.649913 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.649890 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:47.650692 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.650761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12901fb7-1801-4666-a17b-cc1d15663480-hosts-file\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.650761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-socket-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.650761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-log-socket\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.650851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-netd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.650851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-script-lib\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.650851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.650851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650829 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-config\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-os-release\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22f74\" (UniqueName: \"kubernetes.io/projected/db5b0a05-facd-4635-9817-50ca8be76cf5-kube-api-access-22f74\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-kubelet\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-netns\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-system-cni-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.650982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-system-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-multus\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-hostroot\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-registration-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/fee55906-e037-42f7-81a5-1b4e50d4260b-kube-api-access-7d7lj\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651069 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-conf\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-socket-dir-parent\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651156 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-cnibin\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651223 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee55906-e037-42f7-81a5-1b4e50d4260b-ovn-node-metrics-cert\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651241 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651271 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24gb\" (UniqueName: \"kubernetes.io/projected/c7547c86-a7e4-4555-9aa2-08919e5843b8-kube-api-access-s24gb\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651279 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vb4v5\"" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651303 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysconfig\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-sys\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-host\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-os-release\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-env-overrides\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651520 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-tuned\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-kubelet\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-conf-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651603 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-multus-certs\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651613 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-var-lib-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 12:02:47.651668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651703 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbpf\" (UniqueName: \"kubernetes.io/projected/e2107a85-7439-4282-b45c-2f67d06a9489-kube-api-access-6kbpf\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-bin\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-run\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-multus-daemon-config\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndd8\" (UniqueName: \"kubernetes.io/projected/26ea82b2-3bbe-42e1-a642-192b6d32f623-kube-api-access-cndd8\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-modprobe-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12901fb7-1801-4666-a17b-cc1d15663480-tmp-dir\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651887 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651909 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-systemd-units\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-systemd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.651985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-ovn\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9z2h\" (UniqueName: \"kubernetes.io/projected/eb1888fd-c8d8-46bd-a59f-23c2772df11f-kube-api-access-h9z2h\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652046 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652077 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-node-log\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-kubernetes\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-systemd\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-lib-modules\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-var-lib-kubelet\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-tmp\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652232 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652260 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-netns\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-bin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-sys-fs\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-cnibin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652362 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652376 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-device-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-etc-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpfs\" (UniqueName: \"kubernetes.io/projected/12901fb7-1801-4666-a17b-cc1d15663480-kube-api-access-tkpfs\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.652862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-slash\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.653360 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-etc-kubernetes\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.653360 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-cni-binary-copy\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.653360 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.652522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-k8s-cni-cncf-io\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.653710 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.653692 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 12:02:47.653789 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.653769 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jcxq9\"" Jul 09 12:02:47.653900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.653884 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 12:02:47.657039 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.657020 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 12:02:47.684424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.684405 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k7shd" Jul 09 12:02:47.692832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.692787 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k7shd" Jul 09 12:02:47.740976 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.740945 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 12:02:47.752998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.752957 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnjt\" (UniqueName: \"kubernetes.io/projected/cd250024-686c-49f0-a9a1-28a07465bc0b-kube-api-access-wjnjt\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.752998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.752997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5235ece-eae9-41ae-923f-642662767655-konnectivity-ca\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-cnibin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-device-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-etc-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753114 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153f22bc-d128-46ad-bb76-7ee70144b64b-serviceca\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpfs\" (UniqueName: \"kubernetes.io/projected/12901fb7-1801-4666-a17b-cc1d15663480-kube-api-access-tkpfs\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-slash\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753209 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753199 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd250024-686c-49f0-a9a1-28a07465bc0b-host-slash\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-etc-kubernetes\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-cni-binary-copy\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-k8s-cni-cncf-io\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753281 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12901fb7-1801-4666-a17b-cc1d15663480-hosts-file\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753318 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-socket-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-log-socket\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-netd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-script-lib\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753410 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-config\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753440 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-os-release\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22f74\" (UniqueName: \"kubernetes.io/projected/db5b0a05-facd-4635-9817-50ca8be76cf5-kube-api-access-22f74\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-kubelet\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-netns\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-system-cni-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.753654 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753566 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-system-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-multus\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-hostroot\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-registration-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/fee55906-e037-42f7-81a5-1b4e50d4260b-kube-api-access-7d7lj\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-conf\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753710 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-socket-dir-parent\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5235ece-eae9-41ae-923f-642662767655-agent-certs\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-cnibin\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753833 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee55906-e037-42f7-81a5-1b4e50d4260b-ovn-node-metrics-cert\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24gb\" (UniqueName: \"kubernetes.io/projected/c7547c86-a7e4-4555-9aa2-08919e5843b8-kube-api-access-s24gb\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysconfig\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-sys\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753960 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-host\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.754472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.753985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-kubelet-config\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-os-release\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-env-overrides\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-tuned\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-kubelet\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754107 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-conf-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-multus-certs\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-var-lib-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153f22bc-d128-46ad-bb76-7ee70144b64b-host\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dfx\" (UniqueName: \"kubernetes.io/projected/153f22bc-d128-46ad-bb76-7ee70144b64b-kube-api-access-m2dfx\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbpf\" (UniqueName: \"kubernetes.io/projected/e2107a85-7439-4282-b45c-2f67d06a9489-kube-api-access-6kbpf\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754263 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-bin\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754294 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-run\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-multus-daemon-config\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.755276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndd8\" (UniqueName: \"kubernetes.io/projected/26ea82b2-3bbe-42e1-a642-192b6d32f623-kube-api-access-cndd8\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-modprobe-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12901fb7-1801-4666-a17b-cc1d15663480-tmp-dir\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754479 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-systemd-units\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754503 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-systemd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-ovn\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754575 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9z2h\" (UniqueName: \"kubernetes.io/projected/eb1888fd-c8d8-46bd-a59f-23c2772df11f-kube-api-access-h9z2h\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754597 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd250024-686c-49f0-a9a1-28a07465bc0b-iptables-alerter-script\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-node-log\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-kubernetes\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-systemd\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754740 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-lib-modules\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-var-lib-kubelet\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-tmp\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.756084 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-dbus\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-netns\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-bin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.754911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-sys-fs\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755049 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-sys-fs\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755111 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-cnibin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755157 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755208 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-device-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755241 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-etc-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755265 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755418 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-os-release\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755467 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-slash\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.755551 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-etc-kubernetes\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756144 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-env-overrides\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756202 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-cni-binary-copy\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756273 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-k8s-cni-cncf-io\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.756284 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:47.756859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756336 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.756351 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:48.2563294 +0000 UTC m=+3.019054927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756427 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12901fb7-1801-4666-a17b-cc1d15663480-hosts-file\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756519 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-socket-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756621 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-kubelet\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756636 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-netd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756683 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-log-socket\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756739 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-conf-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756792 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-multus-certs\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756837 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.756855 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-var-lib-openvswitch\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757154 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-script-lib\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757288 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757333 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757536 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-os-release\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757631 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee55906-e037-42f7-81a5-1b4e50d4260b-ovnkube-config\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757686 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-socket-dir-parent\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.757799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757747 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757783 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-cnibin\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757898 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-kubelet\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757907 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.757991 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-run-netns\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758039 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db5b0a05-facd-4635-9817-50ca8be76cf5-system-cni-dir\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758094 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-system-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758140 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758152 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-multus\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758201 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-run\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758208 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-registration-dir\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758236 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-hostroot\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758321 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-sys\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758353 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysconfig\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758391 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-host\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758447 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-multus-cni-dir\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.758523 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-host-cni-bin\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761159 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-sysctl-conf\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.764416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761166 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2107a85-7439-4282-b45c-2f67d06a9489-multus-daemon-config\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761560 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-tuned\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761592 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee55906-e037-42f7-81a5-1b4e50d4260b-ovn-node-metrics-cert\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761731 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-systemd\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761784 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-node-log\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.761848 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-kubernetes\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762008 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-lib-modules\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762106 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-var-lib-kubelet\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762283 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ea82b2-3bbe-42e1-a642-192b6d32f623-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762648 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb1888fd-c8d8-46bd-a59f-23c2772df11f-etc-modprobe-d\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762810 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-run-netns\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.762956 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12901fb7-1801-4666-a17b-cc1d15663480-tmp-dir\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.763562 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db5b0a05-facd-4635-9817-50ca8be76cf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.763684 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-ovn\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.763734 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-systemd-units\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.763777 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee55906-e037-42f7-81a5-1b4e50d4260b-run-systemd\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.763896 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1888fd-c8d8-46bd-a59f-23c2772df11f-tmp\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.764097 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2107a85-7439-4282-b45c-2f67d06a9489-host-var-lib-cni-bin\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.765311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.764981 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpfs\" (UniqueName: \"kubernetes.io/projected/12901fb7-1801-4666-a17b-cc1d15663480-kube-api-access-tkpfs\") pod \"node-resolver-bjdxm\" (UID: \"12901fb7-1801-4666-a17b-cc1d15663480\") " pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.769494 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.769400 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22f74\" (UniqueName: \"kubernetes.io/projected/db5b0a05-facd-4635-9817-50ca8be76cf5-kube-api-access-22f74\") pod \"multus-additional-cni-plugins-6zv7s\" (UID: \"db5b0a05-facd-4635-9817-50ca8be76cf5\") " pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.770449 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.770426 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:02:47.770536 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.770454 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:02:47.770536 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.770469 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:47.770536 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.770531 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:48.270512268 +0000 UTC m=+3.033237799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:47.771040 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.771002 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbpf\" (UniqueName: \"kubernetes.io/projected/e2107a85-7439-4282-b45c-2f67d06a9489-kube-api-access-6kbpf\") pod \"multus-vs42f\" (UID: \"e2107a85-7439-4282-b45c-2f67d06a9489\") " pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.772560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.772516 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7lj\" (UniqueName: \"kubernetes.io/projected/fee55906-e037-42f7-81a5-1b4e50d4260b-kube-api-access-7d7lj\") pod \"ovnkube-node-qt4h5\" (UID: \"fee55906-e037-42f7-81a5-1b4e50d4260b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.773836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.773798 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24gb\" (UniqueName: \"kubernetes.io/projected/c7547c86-a7e4-4555-9aa2-08919e5843b8-kube-api-access-s24gb\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:47.775121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.775100 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9z2h\" (UniqueName: \"kubernetes.io/projected/eb1888fd-c8d8-46bd-a59f-23c2772df11f-kube-api-access-h9z2h\") pod \"tuned-fmr9l\" (UID: \"eb1888fd-c8d8-46bd-a59f-23c2772df11f\") " pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.777145 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.777120 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndd8\" (UniqueName: \"kubernetes.io/projected/26ea82b2-3bbe-42e1-a642-192b6d32f623-kube-api-access-cndd8\") pod \"aws-ebs-csi-driver-node-7gp5s\" (UID: \"26ea82b2-3bbe-42e1-a642-192b6d32f623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.846897 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.846839 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" event={"ID":"89a0e5a37365a0cf71513576d558545d","Type":"ContainerStarted","Data":"807a481840bd296aa415f60b45e78dcc75a7442537fe1aabf9024390c415a4ab"} Jul 09 12:02:47.848234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.848204 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" event={"ID":"e2bff7cbf82451b4333adfc3db2c0112","Type":"ContainerStarted","Data":"2595d79eea1e89bb8bfed6520822bd8de9d3368a7de5f8a6bf92471152f1c8d9"} Jul 09 12:02:47.856270 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd250024-686c-49f0-a9a1-28a07465bc0b-iptables-alerter-script\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.856463 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-dbus\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.856463 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnjt\" (UniqueName: \"kubernetes.io/projected/cd250024-686c-49f0-a9a1-28a07465bc0b-kube-api-access-wjnjt\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.856575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5235ece-eae9-41ae-923f-642662767655-konnectivity-ca\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.856575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153f22bc-d128-46ad-bb76-7ee70144b64b-serviceca\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.856575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd250024-686c-49f0-a9a1-28a07465bc0b-host-slash\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.856746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.856746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5235ece-eae9-41ae-923f-642662767655-agent-certs\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.856746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856652 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-kubelet-config\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.856746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153f22bc-d128-46ad-bb76-7ee70144b64b-host\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.856746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.856714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dfx\" (UniqueName: \"kubernetes.io/projected/153f22bc-d128-46ad-bb76-7ee70144b64b-kube-api-access-m2dfx\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.857579 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.857554 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd250024-686c-49f0-a9a1-28a07465bc0b-iptables-alerter-script\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.857742 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.857724 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-dbus\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.858432 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.858394 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5235ece-eae9-41ae-923f-642662767655-konnectivity-ca\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.858818 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.858789 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153f22bc-d128-46ad-bb76-7ee70144b64b-serviceca\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.858906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.858853 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd250024-686c-49f0-a9a1-28a07465bc0b-host-slash\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.859044 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.858947 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:47.859044 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:47.859002 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:48.358982747 +0000 UTC m=+3.121708275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:47.859516 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.859414 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153f22bc-d128-46ad-bb76-7ee70144b64b-host\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.859516 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.859480 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9519607a-8be8-4d08-839c-eb1c0ac80f01-kubelet-config\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:47.862165 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.862122 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5235ece-eae9-41ae-923f-642662767655-agent-certs\") pod \"konnectivity-agent-q9bb6\" (UID: \"f5235ece-eae9-41ae-923f-642662767655\") " pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.872063 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.871975 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dfx\" (UniqueName: \"kubernetes.io/projected/153f22bc-d128-46ad-bb76-7ee70144b64b-kube-api-access-m2dfx\") pod \"node-ca-88pcx\" (UID: \"153f22bc-d128-46ad-bb76-7ee70144b64b\") " pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.873626 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.873602 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnjt\" (UniqueName: \"kubernetes.io/projected/cd250024-686c-49f0-a9a1-28a07465bc0b-kube-api-access-wjnjt\") pod \"iptables-alerter-4mxht\" (UID: \"cd250024-686c-49f0-a9a1-28a07465bc0b\") " pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.943786 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.943751 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:02:47.949588 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.949563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bjdxm" Jul 09 12:02:47.956801 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.956776 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" Jul 09 12:02:47.958306 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:47.958272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee55906_e037_42f7_81a5_1b4e50d4260b.slice/crio-23614bb4b7207e527d94dd1e2cb15ad7b7b7e2b3cd8f19a99e83431d1cd97a9c WatchSource:0}: Error finding container 23614bb4b7207e527d94dd1e2cb15ad7b7b7e2b3cd8f19a99e83431d1cd97a9c: Status 404 returned error can't find the container with id 23614bb4b7207e527d94dd1e2cb15ad7b7b7e2b3cd8f19a99e83431d1cd97a9c Jul 09 12:02:47.958827 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:47.958794 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12901fb7_1801_4666_a17b_cc1d15663480.slice/crio-7a022c510bbb1b6c6ca65698e0facf626119fc504a18bcace3f3f14a675c7060 WatchSource:0}: Error finding container 7a022c510bbb1b6c6ca65698e0facf626119fc504a18bcace3f3f14a675c7060: Status 404 returned error can't find the container with id 7a022c510bbb1b6c6ca65698e0facf626119fc504a18bcace3f3f14a675c7060 Jul 09 12:02:47.961968 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.961762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vs42f" Jul 09 12:02:47.967885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.967857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" Jul 09 12:02:47.968953 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:47.968883 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5b0a05_facd_4635_9817_50ca8be76cf5.slice/crio-2a616aa34c84332b98bf401813951d0a21e1bb0da9b8df01e6aafd856e1a9046 WatchSource:0}: Error finding container 2a616aa34c84332b98bf401813951d0a21e1bb0da9b8df01e6aafd856e1a9046: Status 404 returned error can't find the container with id 2a616aa34c84332b98bf401813951d0a21e1bb0da9b8df01e6aafd856e1a9046 Jul 09 12:02:47.973792 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.973046 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" Jul 09 12:02:47.978623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.977633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88pcx" Jul 09 12:02:47.983985 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.983963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mxht" Jul 09 12:02:47.987499 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:47.987402 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ea82b2_3bbe_42e1_a642_192b6d32f623.slice/crio-261980a9f4014f4b835d9b9226e01867fb0cbef7f5d22656abc4aa818c1531a8 WatchSource:0}: Error finding container 261980a9f4014f4b835d9b9226e01867fb0cbef7f5d22656abc4aa818c1531a8: Status 404 returned error can't find the container with id 261980a9f4014f4b835d9b9226e01867fb0cbef7f5d22656abc4aa818c1531a8 Jul 09 12:02:47.989345 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:47.989214 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:02:47.989667 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:47.989646 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153f22bc_d128_46ad_bb76_7ee70144b64b.slice/crio-272c048f51ea0a55c0ec6219c66aaf470ba8652eb1af553598235f0b98701f92 WatchSource:0}: Error finding container 272c048f51ea0a55c0ec6219c66aaf470ba8652eb1af553598235f0b98701f92: Status 404 returned error can't find the container with id 272c048f51ea0a55c0ec6219c66aaf470ba8652eb1af553598235f0b98701f92 Jul 09 12:02:48.002941 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:48.002750 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd250024_686c_49f0_a9a1_28a07465bc0b.slice/crio-3eba103666cca7d9f87b15c84410af519079dde5c9b21c3d674d41a576ffb84b WatchSource:0}: Error finding container 3eba103666cca7d9f87b15c84410af519079dde5c9b21c3d674d41a576ffb84b: Status 404 returned error can't find the container with id 3eba103666cca7d9f87b15c84410af519079dde5c9b21c3d674d41a576ffb84b Jul 09 12:02:48.007276 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:02:48.006863 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5235ece_eae9_41ae_923f_642662767655.slice/crio-12bb2ad458a87d7ab6c66ceab9bcaa55dde9ebdb12d37f6fb97fc156a1d552a3 WatchSource:0}: Error finding container 12bb2ad458a87d7ab6c66ceab9bcaa55dde9ebdb12d37f6fb97fc156a1d552a3: Status 404 returned error can't find the container with id 12bb2ad458a87d7ab6c66ceab9bcaa55dde9ebdb12d37f6fb97fc156a1d552a3 Jul 09 12:02:48.143538 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.143454 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 12:02:48.260967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.260919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:48.261167 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.261062 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:48.261167 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.261125 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:49.261104767 +0000 UTC m=+4.023830294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:48.362054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.362009 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.362106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362160 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362205 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362219 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:49.362201154 +0000 UTC m=+4.124926685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362221 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:02:48.362241 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362237 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:48.362573 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.362270 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:49.362259919 +0000 UTC m=+4.124985453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:48.695484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.695446 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 11:57:47 +0000 UTC" deadline="2028-04-10 12:17:55.070811082 +0000 UTC" Jul 09 12:02:48.695484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.695479 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15384h15m6.37533578s" Jul 09 12:02:48.842054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.841328 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:48.842054 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.841454 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:48.842054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.841539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:48.842054 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:48.841635 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:48.851400 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.851350 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerStarted","Data":"2a616aa34c84332b98bf401813951d0a21e1bb0da9b8df01e6aafd856e1a9046"} Jul 09 12:02:48.852898 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.852866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" event={"ID":"eb1888fd-c8d8-46bd-a59f-23c2772df11f","Type":"ContainerStarted","Data":"b4c47726e50eb90c0d63a2ed1e8567197d0f76c0169e3c5d8a165ff40e59de9b"} Jul 09 12:02:48.854328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.854304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bjdxm" event={"ID":"12901fb7-1801-4666-a17b-cc1d15663480","Type":"ContainerStarted","Data":"7a022c510bbb1b6c6ca65698e0facf626119fc504a18bcace3f3f14a675c7060"} Jul 09 12:02:48.855532 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.855511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"23614bb4b7207e527d94dd1e2cb15ad7b7b7e2b3cd8f19a99e83431d1cd97a9c"} Jul 09 12:02:48.856967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.856898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q9bb6" event={"ID":"f5235ece-eae9-41ae-923f-642662767655","Type":"ContainerStarted","Data":"12bb2ad458a87d7ab6c66ceab9bcaa55dde9ebdb12d37f6fb97fc156a1d552a3"} Jul 09 12:02:48.858381 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.858358 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mxht" event={"ID":"cd250024-686c-49f0-a9a1-28a07465bc0b","Type":"ContainerStarted","Data":"3eba103666cca7d9f87b15c84410af519079dde5c9b21c3d674d41a576ffb84b"} Jul 09 12:02:48.859843 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.859776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88pcx" event={"ID":"153f22bc-d128-46ad-bb76-7ee70144b64b","Type":"ContainerStarted","Data":"272c048f51ea0a55c0ec6219c66aaf470ba8652eb1af553598235f0b98701f92"} Jul 09 12:02:48.861281 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.861248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" event={"ID":"26ea82b2-3bbe-42e1-a642-192b6d32f623","Type":"ContainerStarted","Data":"261980a9f4014f4b835d9b9226e01867fb0cbef7f5d22656abc4aa818c1531a8"} Jul 09 12:02:48.862562 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:48.862537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vs42f" event={"ID":"e2107a85-7439-4282-b45c-2f67d06a9489","Type":"ContainerStarted","Data":"9dfb3395927d54e2dfd8a93882564a1f850c659ed00d5280c592f3f49e901764"} Jul 09 12:02:49.267906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.267821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:49.268091 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.268050 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:49.268149 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.268118 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:51.268099558 +0000 UTC m=+6.030825077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:49.369126 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.369086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:49.369320 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.369174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:49.377161 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.377126 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:02:49.377161 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.377166 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:02:49.377393 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.377189 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:49.377393 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.377292 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:51.377262024 +0000 UTC m=+6.139987624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:49.377969 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.377946 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:49.378061 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.378022 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:51.378004859 +0000 UTC m=+6.140730375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:49.696048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.695984 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 11:57:47 +0000 UTC" deadline="2028-02-01 11:32:55.660487055 +0000 UTC" Jul 09 12:02:49.696048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.696046 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13727h30m5.964444926s" Jul 09 12:02:49.845071 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.844446 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:49.845071 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:49.844576 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:49.884304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:49.884264 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" event={"ID":"e2bff7cbf82451b4333adfc3db2c0112","Type":"ContainerStarted","Data":"df54fd4898d51ff10c000ad20aefb44a435a6397618565b2b181d2b30c967cc7"} Jul 09 12:02:50.841284 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:50.841202 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:50.841891 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:50.841325 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:50.841891 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:50.841698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:50.841891 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:50.841811 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:50.907049 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:50.905830 2567 generic.go:356] "Generic (PLEG): container finished" podID="89a0e5a37365a0cf71513576d558545d" containerID="39256c6e56ead63c6c59b2f8e09535e89224c6928808af1cf8b0e5b80bdf8ed2" exitCode=0 Jul 09 12:02:50.907049 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:50.906778 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" event={"ID":"89a0e5a37365a0cf71513576d558545d","Type":"ContainerDied","Data":"39256c6e56ead63c6c59b2f8e09535e89224c6928808af1cf8b0e5b80bdf8ed2"} Jul 09 12:02:50.921991 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:50.921903 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.ec2.internal" podStartSLOduration=3.921881968 podStartE2EDuration="3.921881968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:02:49.901494684 +0000 UTC m=+4.664220236" watchObservedRunningTime="2026-07-09 12:02:50.921881968 +0000 UTC m=+5.684607620" Jul 09 12:02:51.289499 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.289453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:51.289692 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.289612 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:51.289692 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.289675 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:55.289654858 +0000 UTC m=+10.052380373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:51.390449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.390360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:51.390449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.390437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:51.390638 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.390574 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:51.390672 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.390641 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:55.390620237 +0000 UTC m=+10.153345768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:51.391102 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.391081 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:02:51.391192 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.391107 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:02:51.391192 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.391121 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:51.391192 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.391169 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:02:55.391153298 +0000 UTC m=+10.153878835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:51.841575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.841451 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:51.842016 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:51.841641 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:51.911567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.911529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" event={"ID":"89a0e5a37365a0cf71513576d558545d","Type":"ContainerStarted","Data":"3f244d54a938f907c2286d261ec2d3869cca1fa5ac50bfe5c8d88c5736325cba"} Jul 09 12:02:51.929443 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:51.929133 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.ec2.internal" podStartSLOduration=4.929115579 podStartE2EDuration="4.929115579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:02:51.928923327 +0000 UTC m=+6.691648882" watchObservedRunningTime="2026-07-09 12:02:51.929115579 +0000 UTC m=+6.691841114" Jul 09 12:02:52.841524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:52.841482 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:52.841725 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:52.841630 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:52.842144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:52.842062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:52.842185 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:52.842142 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:53.844347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:53.843767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:53.844347 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:53.843943 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:54.842074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:54.842039 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:54.842238 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:54.842084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:54.842238 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:54.842185 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:54.842349 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:54.842306 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:55.323082 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:55.323035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:55.323530 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.323184 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:55.323530 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.323267 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:03.323246798 +0000 UTC m=+18.085972327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:02:55.423751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:55.423712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:55.423941 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:55.423786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:55.424010 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.423946 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:55.424047 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.424008 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:03.423991171 +0000 UTC m=+18.186716699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:02:55.424230 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.424208 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:02:55.424317 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.424234 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:02:55.424317 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.424247 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:55.424402 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.424342 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:03.424286965 +0000 UTC m=+18.187012497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:02:55.843017 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:55.842977 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:55.843193 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:55.843108 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:56.841875 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:56.841835 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:56.842316 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:56.841989 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:56.842431 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:56.842413 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:56.842548 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:56.842524 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:57.842208 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:57.842168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:57.842791 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:57.842319 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:58.845613 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:58.843876 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:02:58.845613 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:58.844041 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:02:58.845613 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:58.844449 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:02:58.845613 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:58.844555 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:02:59.841426 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.841217 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:02:59.841582 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:02:59.841517 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:02:59.936095 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.935657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q9bb6" event={"ID":"f5235ece-eae9-41ae-923f-642662767655","Type":"ContainerStarted","Data":"8d95ac7a6e53c091b0db197dbb7ef4b7151e1156dee11bc9147bf132392b695e"} Jul 09 12:02:59.938486 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.938017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88pcx" event={"ID":"153f22bc-d128-46ad-bb76-7ee70144b64b","Type":"ContainerStarted","Data":"d7425ed04ca003c5b1cb74be180a6903f80be09852793d02ff404e265bb299c0"} Jul 09 12:02:59.939865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.939839 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" event={"ID":"26ea82b2-3bbe-42e1-a642-192b6d32f623","Type":"ContainerStarted","Data":"7cb6f2887b17ca88de4dd22c7ae08ca4cf46660ae7b02cba5cc1fa36f63b7bb0"} Jul 09 12:02:59.941607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.941580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerStarted","Data":"c815fcc0262cf3c6d5302da15fe77887d73bb1d20005edf512780766cdfb283f"} Jul 09 12:02:59.943202 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.943172 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bjdxm" event={"ID":"12901fb7-1801-4666-a17b-cc1d15663480","Type":"ContainerStarted","Data":"543f1b02526cad4c88d903ced70551c15213aefc7420824a738b6c4b37b45084"} Jul 09 12:02:59.950919 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.950864 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q9bb6" podStartSLOduration=3.188399713 podStartE2EDuration="13.950848374s" totalImagesPullingTime="10.762448661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:02:59.950143134 +0000 UTC m=+14.712868671" watchObservedRunningTime="2026-07-09 12:02:59.950848374 +0000 UTC m=+14.713573911" Jul 09 12:02:59.965588 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.965549 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bjdxm" podStartSLOduration=4.15203424 podStartE2EDuration="14.965539408s" totalImagesPullingTime="10.813505168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:02:59.964902233 +0000 UTC m=+14.727627769" watchObservedRunningTime="2026-07-09 12:02:59.965539408 +0000 UTC m=+14.728264943" Jul 09 12:02:59.983866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:02:59.983817 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-88pcx" podStartSLOduration=3.200285033 podStartE2EDuration="13.983800737s" totalImagesPullingTime="10.783515704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:02:59.983589418 +0000 UTC m=+14.746314980" watchObservedRunningTime="2026-07-09 12:02:59.983800737 +0000 UTC m=+14.746526275" Jul 09 12:03:00.841485 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:00.841450 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:00.841655 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:00.841493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:00.841655 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:00.841579 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:00.841768 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:00.841742 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:01.841919 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:01.841881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:01.842365 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:01.842046 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:02.841149 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:02.841108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:02.841326 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:02.841107 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:02.841326 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:02.841256 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:02.841416 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:02.841328 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:03.296128 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.296098 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:03:03.296773 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.296735 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:03:03.393465 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.393430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:03.393652 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.393558 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:03:03.393652 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.393612 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:19.393594865 +0000 UTC m=+34.156320379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:03:03.494412 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.494371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:03.494593 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.494436 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:03.494593 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.494568 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:03:03.494712 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.494618 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:19.494603754 +0000 UTC m=+34.257329268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:03:03.495007 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.494985 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:03:03.495007 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.495007 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:03:03.495209 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.495020 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:03:03.495209 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.495063 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:19.495049525 +0000 UTC m=+34.257775056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:03:03.841424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.841395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:03.841597 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:03.841527 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:03.953996 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.953956 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="c815fcc0262cf3c6d5302da15fe77887d73bb1d20005edf512780766cdfb283f" exitCode=0 Jul 09 12:03:03.954178 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:03.954006 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"c815fcc0262cf3c6d5302da15fe77887d73bb1d20005edf512780766cdfb283f"} Jul 09 12:03:04.841109 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:04.841077 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:04.841591 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:04.841544 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:04.841999 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:04.841970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:04.842144 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:04.842131 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:04.957662 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:04.957624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mxht" event={"ID":"cd250024-686c-49f0-a9a1-28a07465bc0b","Type":"ContainerStarted","Data":"660f0ba3a17b79993ef641efdad3cf23b0309d0eea5155d6ec07217226da91d5"} Jul 09 12:03:04.972474 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:04.972419 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4mxht" podStartSLOduration=8.213793604 podStartE2EDuration="18.972399901s" totalImagesPullingTime="10.758606297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:04.972202163 +0000 UTC m=+19.734927699" watchObservedRunningTime="2026-07-09 12:03:04.972399901 +0000 UTC m=+19.735125438" Jul 09 12:03:05.842716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:05.842686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:05.843219 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:05.842782 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:06.842010 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:06.841967 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:06.842191 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:06.842098 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:06.842191 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:06.842156 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:06.842317 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:06.842265 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:07.822033 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.821625 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 12:03:07.843605 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.843581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:07.843723 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:07.843702 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:07.965061 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.965028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" event={"ID":"eb1888fd-c8d8-46bd-a59f-23c2772df11f","Type":"ContainerStarted","Data":"04c15c459a7f3785b6b14b37a67a019abc800dc44ca95a86db69985156eb0f39"} Jul 09 12:03:07.967564 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.967532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"03999db514c90403849722a74be22309b6903f6c2933386445b09d0a0e1a0050"} Jul 09 12:03:07.967667 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.967621 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"f6326c152475fac0723fd83c146667f685e73769a7be70137d6a4d77d7045eca"} Jul 09 12:03:07.967730 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.967675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"1788bc64d4f78d7e2e181ef776538ecfbeda08c6e78917764ffebabb91580504"} Jul 09 12:03:07.967730 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.967722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"ed39aade4749d64f8de2644445929050cd44b299860662fb98be529e69e15858"} Jul 09 12:03:07.969446 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.969420 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" event={"ID":"26ea82b2-3bbe-42e1-a642-192b6d32f623","Type":"ContainerStarted","Data":"8afd92190889be4d5a35f5ca2fe79a9be2001dcf0c7ab898d7786f9156c0d9b5"} Jul 09 12:03:07.970746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.970726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vs42f" event={"ID":"e2107a85-7439-4282-b45c-2f67d06a9489","Type":"ContainerStarted","Data":"91ae18b2f3e21e80e62ed53ef3bc1e4b447e75c8029af4a1902bf597ba2d191d"} Jul 09 12:03:07.981635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.981582 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fmr9l" podStartSLOduration=11.169174919 podStartE2EDuration="21.981569498s" totalImagesPullingTime="10.812394579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:07.981078736 +0000 UTC m=+22.743804271" watchObservedRunningTime="2026-07-09 12:03:07.981569498 +0000 UTC m=+22.744295034" Jul 09 12:03:07.997504 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:07.997456 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vs42f" podStartSLOduration=3.530311131 podStartE2EDuration="22.997441343s" totalImagesPullingTime="19.467130212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:07.997339676 +0000 UTC m=+22.760065240" watchObservedRunningTime="2026-07-09 12:03:07.997441343 +0000 UTC m=+22.760166878" Jul 09 12:03:08.711783 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.711618 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T12:03:07.822033638Z","UUID":"48b4dd43-3417-4ce6-8b4b-c26dcb3e3957","Handler":null,"Name":"","Endpoint":""} Jul 09 12:03:08.713361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.713339 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 12:03:08.713503 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.713385 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 12:03:08.841480 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.841448 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:08.842115 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.841491 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:08.842115 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:08.841557 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:08.842115 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:08.841675 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:08.976108 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.976025 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"22c8387862f9c5f7d204d9be27b8be9ce82d3b9beac4aedb959495749c880e6a"} Jul 09 12:03:08.976108 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:08.976105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"e46ab222a2c0e09d29e99c7d84ce271fc55e2f3686b32047a6190480ca03df19"} Jul 09 12:03:09.841317 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:09.841280 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:09.841493 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:09.841422 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:09.980530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:09.980489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" event={"ID":"26ea82b2-3bbe-42e1-a642-192b6d32f623","Type":"ContainerStarted","Data":"1ffcb5c4a451f740216574273418271567973b79084e11ae8f68d6373e9621ed"} Jul 09 12:03:09.998686 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:09.998637 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gp5s" podStartSLOduration=3.22805587 podStartE2EDuration="23.998618267s" totalImagesPullingTime="20.770562397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:09.99854114 +0000 UTC m=+24.761266676" watchObservedRunningTime="2026-07-09 12:03:09.998618267 +0000 UTC m=+24.761343804" Jul 09 12:03:10.841355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:10.841317 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:10.841355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:10.841342 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:10.841684 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:10.841454 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:10.841684 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:10.841589 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:10.988645 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:10.988588 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"7d34f29d01856497389f8f4780038cfd337dfbb8dfa0790019e5143527e78a39"} Jul 09 12:03:11.842038 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:11.842004 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:11.842207 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:11.842118 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:12.842117 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.841849 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:12.842642 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.841855 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:12.842642 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:12.842171 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:12.842642 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:12.842232 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:12.995431 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.995395 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="54a1d225d64877be8f688486def6ad100f553d7e46ee293ca931e53d05b1c407" exitCode=0 Jul 09 12:03:12.995588 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.995479 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"54a1d225d64877be8f688486def6ad100f553d7e46ee293ca931e53d05b1c407"} Jul 09 12:03:12.999439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.999392 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" event={"ID":"fee55906-e037-42f7-81a5-1b4e50d4260b","Type":"ContainerStarted","Data":"142446d27e4294f2ea90d10f6703e5d60f1f38d16ce76579577735eaa91d3069"} Jul 09 12:03:12.999676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.999655 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:12.999758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.999710 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:12.999758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:12.999742 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:13.016070 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:13.016047 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:13.016207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:13.016141 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:13.067679 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:13.067632 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" podStartSLOduration=8.585947758 podStartE2EDuration="28.067616166s" totalImagesPullingTime="19.481668408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:13.066790893 +0000 UTC m=+27.829516429" watchObservedRunningTime="2026-07-09 12:03:13.067616166 +0000 UTC m=+27.830341702" Jul 09 12:03:13.841805 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:13.841765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:13.842010 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:13.841886 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:14.384073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.383862 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4wkb"] Jul 09 12:03:14.384407 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.384168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:14.384407 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:14.384256 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:14.387452 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.387426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cb4v"] Jul 09 12:03:14.387556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.387542 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:14.387681 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:14.387659 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:14.388164 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.388132 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jwdfr"] Jul 09 12:03:14.388250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:14.388236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:14.388365 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:14.388345 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:15.005700 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:15.005667 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="9f1372c27aa1bd008c7fc8d90415999bd5dc9688365ffdd2fdfb9b49be9955f9" exitCode=0 Jul 09 12:03:15.005848 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:15.005736 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"9f1372c27aa1bd008c7fc8d90415999bd5dc9688365ffdd2fdfb9b49be9955f9"} Jul 09 12:03:15.843267 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:15.843232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:15.843709 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:15.843340 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:15.843709 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:15.843385 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:15.843709 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:15.843379 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:15.843709 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:15.843464 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:15.843709 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:15.843535 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:16.009712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:16.009681 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="9cbd18da70d2bebddab44cbd63f65d86496e446838917377f9fa21e435439830" exitCode=0 Jul 09 12:03:16.009881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:16.009742 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"9cbd18da70d2bebddab44cbd63f65d86496e446838917377f9fa21e435439830"} Jul 09 12:03:17.065255 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.065188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:03:17.065945 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.065371 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 12:03:17.065945 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.065832 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q9bb6" Jul 09 12:03:17.841900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.841865 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:17.842081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.841871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:17.842081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:17.842014 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:17.842081 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:17.842032 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:17.842247 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:17.842091 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:17.842247 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:17.842181 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:19.411049 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.411011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:19.411495 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.411176 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:03:19.411495 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.411239 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs podName:c7547c86-a7e4-4555-9aa2-08919e5843b8 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:51.411224787 +0000 UTC m=+66.173950303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs") pod "network-metrics-daemon-jwdfr" (UID: "c7547c86-a7e4-4555-9aa2-08919e5843b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 12:03:19.511425 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.511385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:19.511653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.511460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:19.511653 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511595 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 12:03:19.511653 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511602 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 12:03:19.511653 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511628 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 12:03:19.511653 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511643 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pfcc6 for pod openshift-network-diagnostics/network-check-target-g4wkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:03:19.511853 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511661 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret podName:9519607a-8be8-4d08-839c-eb1c0ac80f01 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:51.511643421 +0000 UTC m=+66.274368953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret") pod "global-pull-secret-syncer-5cb4v" (UID: "9519607a-8be8-4d08-839c-eb1c0ac80f01") : object "kube-system"/"original-pull-secret" not registered Jul 09 12:03:19.511853 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.511696 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6 podName:4666f02e-dd85-4893-bc39-238b89cb5623 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:51.511680875 +0000 UTC m=+66.274406405 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfcc6" (UniqueName: "kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6") pod "network-check-target-g4wkb" (UID: "4666f02e-dd85-4893-bc39-238b89cb5623") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 12:03:19.841124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.841087 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:19.841371 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.841087 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:19.841371 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.841231 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jwdfr" podUID="c7547c86-a7e4-4555-9aa2-08919e5843b8" Jul 09 12:03:19.841371 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:19.841096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:19.841371 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.841296 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wkb" podUID="4666f02e-dd85-4893-bc39-238b89cb5623" Jul 09 12:03:19.841371 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:19.841357 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cb4v" podUID="9519607a-8be8-4d08-839c-eb1c0ac80f01" Jul 09 12:03:20.063333 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.063300 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-7.ec2.internal" event="NodeReady" Jul 09 12:03:20.063507 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.063450 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 12:03:20.110839 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.110762 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pkx95"] Jul 09 12:03:20.132234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.132198 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qc2mn"] Jul 09 12:03:20.132383 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.132325 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.134556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.134536 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 12:03:20.134684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.134598 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 12:03:20.134742 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.134697 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzdcd\"" Jul 09 12:03:20.158490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.158461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkx95"] Jul 09 12:03:20.158490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.158497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qc2mn"] Jul 09 12:03:20.158655 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.158577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.160687 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.160660 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7t5w9\"" Jul 09 12:03:20.160839 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.160668 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 12:03:20.160839 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.160771 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 12:03:20.161020 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.160921 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 12:03:20.216339 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.216529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9354067b-2812-4427-b351-ce4b19bb8b97-cert\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.216529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-config-volume\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.216529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqns\" (UniqueName: \"kubernetes.io/projected/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-kube-api-access-fsqns\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.216529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffw2d\" (UniqueName: \"kubernetes.io/projected/9354067b-2812-4427-b351-ce4b19bb8b97-kube-api-access-ffw2d\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.216529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.216443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-tmp-dir\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.317712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9354067b-2812-4427-b351-ce4b19bb8b97-cert\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-config-volume\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqns\" (UniqueName: \"kubernetes.io/projected/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-kube-api-access-fsqns\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffw2d\" (UniqueName: \"kubernetes.io/projected/9354067b-2812-4427-b351-ce4b19bb8b97-kube-api-access-ffw2d\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:20.317842 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.317862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-tmp-dir\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.317911 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:20.317911 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls podName:4c69ba0e-f1cf-4c6e-902d-27afaecf1a49 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:20.817896727 +0000 UTC m=+35.580622240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls") pod "dns-default-pkx95" (UID: "4c69ba0e-f1cf-4c6e-902d-27afaecf1a49") : secret "dns-default-metrics-tls" not found Jul 09 12:03:20.318592 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.318570 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-config-volume\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.318766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.318749 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-tmp-dir\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.322905 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.322717 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9354067b-2812-4427-b351-ce4b19bb8b97-cert\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.335428 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.335407 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqns\" (UniqueName: \"kubernetes.io/projected/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-kube-api-access-fsqns\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.335560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.335430 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffw2d\" (UniqueName: \"kubernetes.io/projected/9354067b-2812-4427-b351-ce4b19bb8b97-kube-api-access-ffw2d\") pod \"ingress-canary-qc2mn\" (UID: \"9354067b-2812-4427-b351-ce4b19bb8b97\") " pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.467484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.467444 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qc2mn" Jul 09 12:03:20.669347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.669315 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qc2mn"] Jul 09 12:03:20.821358 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:20.821228 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:20.821519 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:20.821407 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 12:03:20.821519 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:20.821460 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls podName:4c69ba0e-f1cf-4c6e-902d-27afaecf1a49 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:21.821444657 +0000 UTC m=+36.584170170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls") pod "dns-default-pkx95" (UID: "4c69ba0e-f1cf-4c6e-902d-27afaecf1a49") : secret "dns-default-metrics-tls" not found Jul 09 12:03:21.762438 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.762403 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sxpnh"] Jul 09 12:03:21.797049 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.797006 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxpnh"] Jul 09 12:03:21.797228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.797161 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:21.799423 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.799397 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 12:03:21.799423 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.799417 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 12:03:21.799655 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.799633 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 12:03:21.800321 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.800303 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 12:03:21.800780 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.800761 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvw8k\"" Jul 09 12:03:21.829158 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.829132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:21.829320 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:21.829283 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 12:03:21.829376 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:21.829347 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls podName:4c69ba0e-f1cf-4c6e-902d-27afaecf1a49 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:23.8293298 +0000 UTC m=+38.592055314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls") pod "dns-default-pkx95" (UID: "4c69ba0e-f1cf-4c6e-902d-27afaecf1a49") : secret "dns-default-metrics-tls" not found Jul 09 12:03:21.841586 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.841558 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:21.841724 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.841558 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:21.841963 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.841558 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844100 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844100 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpf42\"" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844142 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844150 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844103 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 12:03:21.844434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.844103 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-62g6t\"" Jul 09 12:03:21.930197 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.930147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:21.930197 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.930201 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-crio-socket\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:21.930421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.930261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989z5\" (UniqueName: \"kubernetes.io/projected/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-api-access-989z5\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:21.930421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.930324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-data-volume\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:21.930421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:21.930400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031176 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-crio-socket\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989z5\" (UniqueName: \"kubernetes.io/projected/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-api-access-989z5\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031481 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.031286 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 12:03:22.031481 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-data-volume\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031481 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.031376 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:22.531356068 +0000 UTC m=+37.294081586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : configmap "kube-rbac-proxy" not found Jul 09 12:03:22.031481 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031419 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-crio-socket\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031481 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.031725 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.031553 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:22.031725 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.031609 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:22.531592976 +0000 UTC m=+37.294318490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:22.031832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.031813 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-data-volume\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.041607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.041580 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989z5\" (UniqueName: \"kubernetes.io/projected/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-api-access-989z5\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.176874 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:22.176831 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9354067b_2812_4427_b351_ce4b19bb8b97.slice/crio-7be73bc802a6c6af35c8644e7d11511774d5a9cb7de6cfb41a197f468086bd65 WatchSource:0}: Error finding container 7be73bc802a6c6af35c8644e7d11511774d5a9cb7de6cfb41a197f468086bd65: Status 404 returned error can't find the container with id 7be73bc802a6c6af35c8644e7d11511774d5a9cb7de6cfb41a197f468086bd65 Jul 09 12:03:22.538292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.538263 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.538427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:22.538316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:22.538427 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.538393 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 12:03:22.538427 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.538416 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:22.538654 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.538450 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:23.538434829 +0000 UTC m=+38.301160345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : configmap "kube-rbac-proxy" not found Jul 09 12:03:22.538654 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:22.538474 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:23.538455988 +0000 UTC m=+38.301181501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:23.030938 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.030757 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="e96838abddca6e7fc9e10cb5274a2f34fd31ecd26c9af64c386b5af0156ce592" exitCode=0 Jul 09 12:03:23.031324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.030822 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"e96838abddca6e7fc9e10cb5274a2f34fd31ecd26c9af64c386b5af0156ce592"} Jul 09 12:03:23.032205 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.032186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qc2mn" event={"ID":"9354067b-2812-4427-b351-ce4b19bb8b97","Type":"ContainerStarted","Data":"7be73bc802a6c6af35c8644e7d11511774d5a9cb7de6cfb41a197f468086bd65"} Jul 09 12:03:23.545435 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.545400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:23.545605 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.545510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:23.545605 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.545524 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 12:03:23.545605 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.545591 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:25.545573241 +0000 UTC m=+40.308298755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : configmap "kube-rbac-proxy" not found Jul 09 12:03:23.545733 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.545613 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:23.545733 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.545648 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:25.54563803 +0000 UTC m=+40.308363547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:23.848676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:23.848594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:23.848849 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.848737 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 12:03:23.848849 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:23.848793 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls podName:4c69ba0e-f1cf-4c6e-902d-27afaecf1a49 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:27.848778974 +0000 UTC m=+42.611504488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls") pod "dns-default-pkx95" (UID: "4c69ba0e-f1cf-4c6e-902d-27afaecf1a49") : secret "dns-default-metrics-tls" not found Jul 09 12:03:24.027371 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.027342 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-t6h4n"] Jul 09 12:03:24.038416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.038359 2567 generic.go:356] "Generic (PLEG): container finished" podID="db5b0a05-facd-4635-9817-50ca8be76cf5" containerID="5f24c91fd71bd91715c346bc6fe3752f050e5aa8f91f5dbce439a8f024cb2710" exitCode=0 Jul 09 12:03:24.044794 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.044769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-t6h4n"] Jul 09 12:03:24.044973 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.044807 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerDied","Data":"5f24c91fd71bd91715c346bc6fe3752f050e5aa8f91f5dbce439a8f024cb2710"} Jul 09 12:03:24.044973 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.044915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.047261 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.047240 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 12:03:24.047261 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.047250 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 12:03:24.047406 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.047303 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 12:03:24.047406 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.047262 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 12:03:24.047540 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.047522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bn6zv\"" Jul 09 12:03:24.151151 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.151124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-key\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.151329 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.151306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6d2\" (UniqueName: \"kubernetes.io/projected/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-kube-api-access-vj6d2\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.151396 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.151361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-cabundle\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.252746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.252710 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-key\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.252894 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.252788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6d2\" (UniqueName: \"kubernetes.io/projected/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-kube-api-access-vj6d2\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.252894 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.252814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-cabundle\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.253648 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.253620 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-cabundle\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.256885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.256856 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-signing-key\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.269007 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.268978 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6d2\" (UniqueName: \"kubernetes.io/projected/1ea2cb94-ff81-4f8f-8650-1a2de29d2565-kube-api-access-vj6d2\") pod \"service-ca-758f599c5-t6h4n\" (UID: \"1ea2cb94-ff81-4f8f-8650-1a2de29d2565\") " pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.356777 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.356736 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-t6h4n" Jul 09 12:03:24.461406 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.461377 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bjdxm_12901fb7-1801-4666-a17b-cc1d15663480/dns-node-resolver/0.log" Jul 09 12:03:24.502190 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:24.500447 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-t6h4n"] Jul 09 12:03:24.506191 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:24.506153 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea2cb94_ff81_4f8f_8650_1a2de29d2565.slice/crio-f32ee9a4f725e1c98ef2fdcb0efad6d3edad74c0686cb596ea401dc71fb37599 WatchSource:0}: Error finding container f32ee9a4f725e1c98ef2fdcb0efad6d3edad74c0686cb596ea401dc71fb37599: Status 404 returned error can't find the container with id f32ee9a4f725e1c98ef2fdcb0efad6d3edad74c0686cb596ea401dc71fb37599 Jul 09 12:03:25.043007 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.042723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-t6h4n" event={"ID":"1ea2cb94-ff81-4f8f-8650-1a2de29d2565","Type":"ContainerStarted","Data":"f32ee9a4f725e1c98ef2fdcb0efad6d3edad74c0686cb596ea401dc71fb37599"} Jul 09 12:03:25.046560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.046536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" event={"ID":"db5b0a05-facd-4635-9817-50ca8be76cf5","Type":"ContainerStarted","Data":"70ff5430e0be5efe2e468edaffe778a21a37eb745d0807608c93d5e5d5c76270"} Jul 09 12:03:25.069435 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.069367 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zv7s" podStartSLOduration=12.313938044 podStartE2EDuration="40.069346845s" totalImagesPullingTime="27.755408801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:25.068407202 +0000 UTC m=+39.831132759" watchObservedRunningTime="2026-07-09 12:03:25.069346845 +0000 UTC m=+39.832072385" Jul 09 12:03:25.450510 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.450478 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88pcx_153f22bc-d128-46ad-bb76-7ee70144b64b/node-ca/0.log" Jul 09 12:03:25.564648 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.564607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:25.564835 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:25.564688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:25.564835 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:25.564791 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:25.564962 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:25.564835 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 12:03:25.564962 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:25.564866 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:29.564844919 +0000 UTC m=+44.327570447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:25.564962 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:25.564915 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:29.564898137 +0000 UTC m=+44.327623674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : configmap "kube-rbac-proxy" not found Jul 09 12:03:26.050705 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:26.050608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qc2mn" event={"ID":"9354067b-2812-4427-b351-ce4b19bb8b97","Type":"ContainerStarted","Data":"a893f2aba879e847a989881e38fe98b1419ab50144618d51eeedb5ae0502a343"} Jul 09 12:03:26.066044 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:26.065986 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qc2mn" podStartSLOduration=2.546481748 podStartE2EDuration="6.065965948s" totalImagesPullingTime="3.5194842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:26.065698659 +0000 UTC m=+40.828424215" watchObservedRunningTime="2026-07-09 12:03:26.065965948 +0000 UTC m=+40.828691486" Jul 09 12:03:26.649579 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:26.649551 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qc2mn_9354067b-2812-4427-b351-ce4b19bb8b97/serve-healthcheck-canary/0.log" Jul 09 12:03:27.885201 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:27.885164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:27.885584 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:27.885324 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 12:03:27.885584 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:27.885385 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls podName:4c69ba0e-f1cf-4c6e-902d-27afaecf1a49 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:35.885370896 +0000 UTC m=+50.648096414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls") pod "dns-default-pkx95" (UID: "4c69ba0e-f1cf-4c6e-902d-27afaecf1a49") : secret "dns-default-metrics-tls" not found Jul 09 12:03:29.060939 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:29.060881 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-t6h4n" event={"ID":"1ea2cb94-ff81-4f8f-8650-1a2de29d2565","Type":"ContainerStarted","Data":"1d8f7370b812de1b317ed47f659803511fe1229e1dd792169893cae7c1afb94c"} Jul 09 12:03:29.078216 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:29.078167 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-t6h4n" podStartSLOduration=1.407871597 podStartE2EDuration="5.078152532s" totalImagesPullingTime="3.670280935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:29.077198304 +0000 UTC m=+43.839923841" watchObservedRunningTime="2026-07-09 12:03:29.078152532 +0000 UTC m=+43.840878068" Jul 09 12:03:29.598008 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:29.597965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:29.598200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:29.598024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:29.598200 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:29.598104 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 12:03:29.598200 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:29.598118 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:29.598200 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:29.598174 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:37.598155711 +0000 UTC m=+52.360881239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:29.598200 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:29.598191 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:37.598182244 +0000 UTC m=+52.360907759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : configmap "kube-rbac-proxy" not found Jul 09 12:03:35.951640 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:35.951597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:35.954830 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:35.954801 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c69ba0e-f1cf-4c6e-902d-27afaecf1a49-metrics-tls\") pod \"dns-default-pkx95\" (UID: \"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49\") " pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:36.043250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:36.043211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:36.187066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:36.187033 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkx95"] Jul 09 12:03:37.081577 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:37.081544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkx95" event={"ID":"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49","Type":"ContainerStarted","Data":"bf27d10ebbe2feeb2d4c8f8b8d349993e49bd0dc022073dc289442230502f01d"} Jul 09 12:03:37.666124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:37.666086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:37.666432 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:37.666153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:37.666432 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:37.666239 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 12:03:37.666432 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:03:37.666312 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls podName:8fbd8b80-f0bf-464e-981b-bef2e22e9f89 nodeName:}" failed. No retries permitted until 2026-07-09 12:03:53.666292227 +0000 UTC m=+68.429017741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxpnh" (UID: "8fbd8b80-f0bf-464e-981b-bef2e22e9f89") : secret "insights-runtime-extractor-tls" not found Jul 09 12:03:37.666789 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:37.666757 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:40.091719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:40.091683 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkx95" event={"ID":"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49","Type":"ContainerStarted","Data":"a408e7b641bfe9bd6ad6bc62a6289f16537a1c1fb5dcf3a19f9eaf56a0af98ab"} Jul 09 12:03:40.092144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:40.091741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkx95" event={"ID":"4c69ba0e-f1cf-4c6e-902d-27afaecf1a49","Type":"ContainerStarted","Data":"99b1bb7550cad1e745b6ef2f213d8dfbf0469b226f5b60380a85b37e6e0408d4"} Jul 09 12:03:40.092144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:40.091823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:40.110542 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:40.110494 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pkx95" podStartSLOduration=17.067523418 podStartE2EDuration="20.110477592s" totalImagesPullingTime="3.042954174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:40.109880516 +0000 UTC m=+54.872606071" watchObservedRunningTime="2026-07-09 12:03:40.110477592 +0000 UTC m=+54.873203128" Jul 09 12:03:45.016956 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:45.016905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt4h5" Jul 09 12:03:50.096520 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:50.096490 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pkx95" Jul 09 12:03:51.462947 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.462886 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:51.465490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.465471 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 12:03:51.475623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.475599 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7547c86-a7e4-4555-9aa2-08919e5843b8-metrics-certs\") pod \"network-metrics-daemon-jwdfr\" (UID: \"c7547c86-a7e4-4555-9aa2-08919e5843b8\") " pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:51.556512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.556484 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-62g6t\"" Jul 09 12:03:51.563907 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.563884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jwdfr" Jul 09 12:03:51.564068 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.564047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:51.564111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.564092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:51.566525 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.566504 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 12:03:51.566621 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.566533 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 12:03:51.576361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.576334 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 12:03:51.577033 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.577010 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9519607a-8be8-4d08-839c-eb1c0ac80f01-original-pull-secret\") pod \"global-pull-secret-syncer-5cb4v\" (UID: \"9519607a-8be8-4d08-839c-eb1c0ac80f01\") " pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:51.587512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.587487 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfcc6\" (UniqueName: \"kubernetes.io/projected/4666f02e-dd85-4893-bc39-238b89cb5623-kube-api-access-pfcc6\") pod \"network-check-target-g4wkb\" (UID: \"4666f02e-dd85-4893-bc39-238b89cb5623\") " pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:51.688038 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.688011 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jwdfr"] Jul 09 12:03:51.691368 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:51.691341 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7547c86_a7e4_4555_9aa2_08919e5843b8.slice/crio-fd892b3f11309daeec8df0e831ece4f2d3698c7ae1f5f5a6145b9facd54b636b WatchSource:0}: Error finding container fd892b3f11309daeec8df0e831ece4f2d3698c7ae1f5f5a6145b9facd54b636b: Status 404 returned error can't find the container with id fd892b3f11309daeec8df0e831ece4f2d3698c7ae1f5f5a6145b9facd54b636b Jul 09 12:03:51.860161 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.860075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cb4v" Jul 09 12:03:51.868857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.868838 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpf42\"" Jul 09 12:03:51.876969 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.876946 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:03:51.995712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:51.995677 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cb4v"] Jul 09 12:03:51.998479 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:51.998451 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9519607a_8be8_4d08_839c_eb1c0ac80f01.slice/crio-66824690d68d55780342dc93918bad73ea44950d1dc425e6dccba71e7c1270db WatchSource:0}: Error finding container 66824690d68d55780342dc93918bad73ea44950d1dc425e6dccba71e7c1270db: Status 404 returned error can't find the container with id 66824690d68d55780342dc93918bad73ea44950d1dc425e6dccba71e7c1270db Jul 09 12:03:52.017750 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:52.017722 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4wkb"] Jul 09 12:03:52.020749 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:52.020722 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4666f02e_dd85_4893_bc39_238b89cb5623.slice/crio-49a4e7ff94c148b57c2abe5f1bbb9559d5a0db5f982e2552b660ae17be903d81 WatchSource:0}: Error finding container 49a4e7ff94c148b57c2abe5f1bbb9559d5a0db5f982e2552b660ae17be903d81: Status 404 returned error can't find the container with id 49a4e7ff94c148b57c2abe5f1bbb9559d5a0db5f982e2552b660ae17be903d81 Jul 09 12:03:52.122270 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:52.122174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4wkb" event={"ID":"4666f02e-dd85-4893-bc39-238b89cb5623","Type":"ContainerStarted","Data":"49a4e7ff94c148b57c2abe5f1bbb9559d5a0db5f982e2552b660ae17be903d81"} Jul 09 12:03:52.123194 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:52.123168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5cb4v" event={"ID":"9519607a-8be8-4d08-839c-eb1c0ac80f01","Type":"ContainerStarted","Data":"66824690d68d55780342dc93918bad73ea44950d1dc425e6dccba71e7c1270db"} Jul 09 12:03:52.124229 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:52.124206 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jwdfr" event={"ID":"c7547c86-a7e4-4555-9aa2-08919e5843b8","Type":"ContainerStarted","Data":"fd892b3f11309daeec8df0e831ece4f2d3698c7ae1f5f5a6145b9facd54b636b"} Jul 09 12:03:53.682906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:53.682818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:53.685999 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:53.685918 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fbd8b80-f0bf-464e-981b-bef2e22e9f89-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxpnh\" (UID: \"8fbd8b80-f0bf-464e-981b-bef2e22e9f89\") " pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:53.911993 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:53.911892 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvw8k\"" Jul 09 12:03:53.920217 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:53.919978 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxpnh" Jul 09 12:03:54.068310 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:54.068272 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxpnh"] Jul 09 12:03:54.071351 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:03:54.071322 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbd8b80_f0bf_464e_981b_bef2e22e9f89.slice/crio-34e479db341ceeb646d606ea70922c47a70826c15312144c7571df31bf194477 WatchSource:0}: Error finding container 34e479db341ceeb646d606ea70922c47a70826c15312144c7571df31bf194477: Status 404 returned error can't find the container with id 34e479db341ceeb646d606ea70922c47a70826c15312144c7571df31bf194477 Jul 09 12:03:54.133807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:54.133765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jwdfr" event={"ID":"c7547c86-a7e4-4555-9aa2-08919e5843b8","Type":"ContainerStarted","Data":"6992b405710c8d512245f9f4502de8f40db70b7563252c5593aff99d39466356"} Jul 09 12:03:54.134029 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:54.133834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jwdfr" event={"ID":"c7547c86-a7e4-4555-9aa2-08919e5843b8","Type":"ContainerStarted","Data":"ea4896f9df8a60f8798edeff44a97a54fb8da2f9e7c4cf0f164af0fe43c3a347"} Jul 09 12:03:54.135096 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:54.135065 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxpnh" event={"ID":"8fbd8b80-f0bf-464e-981b-bef2e22e9f89","Type":"ContainerStarted","Data":"34e479db341ceeb646d606ea70922c47a70826c15312144c7571df31bf194477"} Jul 09 12:03:54.150413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:03:54.150335 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jwdfr" podStartSLOduration=67.282784614 podStartE2EDuration="1m9.150317503s" totalImagesPullingTime="1.867532889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:03:54.150040895 +0000 UTC m=+68.912766454" watchObservedRunningTime="2026-07-09 12:03:54.150317503 +0000 UTC m=+68.913043039" Jul 09 12:04:00.163663 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:00.163625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4wkb" event={"ID":"4666f02e-dd85-4893-bc39-238b89cb5623","Type":"ContainerStarted","Data":"6c1abde40e59ff6c53e0058fa8e9c31cfb21fed534df8907e8a912dc259008b5"} Jul 09 12:04:00.164018 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:00.163777 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:04:00.179077 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:00.179016 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g4wkb" podStartSLOduration=67.159400156 podStartE2EDuration="1m15.178994021s" totalImagesPullingTime="8.019593865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:04:00.178176071 +0000 UTC m=+74.940901625" watchObservedRunningTime="2026-07-09 12:04:00.178994021 +0000 UTC m=+74.941719557" Jul 09 12:04:01.168270 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:01.168227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5cb4v" event={"ID":"9519607a-8be8-4d08-839c-eb1c0ac80f01","Type":"ContainerStarted","Data":"d3a75314b6d5b8afa77894193cc00a72fefac767796fcd1925dd403fe228fa7e"} Jul 09 12:04:01.169990 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:01.169961 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxpnh" event={"ID":"8fbd8b80-f0bf-464e-981b-bef2e22e9f89","Type":"ContainerStarted","Data":"63931b2ef86b94b469c75a1cb9036460524e6a8aaa4d6b1cda972caf1c0407b4"} Jul 09 12:04:01.186433 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:01.186383 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5cb4v" podStartSLOduration=67.133454838 podStartE2EDuration="1m15.186366586s" totalImagesPullingTime="8.052911748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:04:01.186075697 +0000 UTC m=+75.948801258" watchObservedRunningTime="2026-07-09 12:04:01.186366586 +0000 UTC m=+75.949092182" Jul 09 12:04:02.176002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:02.175959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxpnh" event={"ID":"8fbd8b80-f0bf-464e-981b-bef2e22e9f89","Type":"ContainerStarted","Data":"34ea8925d18d08cfa08ae0eb04bb33f1c54f183a810d689f86975f51de59edbf"} Jul 09 12:04:04.186242 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:04.186207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxpnh" event={"ID":"8fbd8b80-f0bf-464e-981b-bef2e22e9f89","Type":"ContainerStarted","Data":"15ab54ff4794215f7c9f0ea83e74aa3a89cace9c9d791f4e98d544d1aeae5908"} Jul 09 12:04:04.204792 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:04.204734 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sxpnh" podStartSLOduration=33.7205888 podStartE2EDuration="43.204716797s" totalImagesPullingTime="9.484127997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:03:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:04:04.203680993 +0000 UTC m=+78.966412400" watchObservedRunningTime="2026-07-09 12:04:04.204716797 +0000 UTC m=+78.967442332" Jul 09 12:04:31.172802 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:31.172765 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g4wkb" Jul 09 12:04:33.070475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:33.070419 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkx95_4c69ba0e-f1cf-4c6e-902d-27afaecf1a49/dns/0.log" Jul 09 12:04:33.270593 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:33.270566 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkx95_4c69ba0e-f1cf-4c6e-902d-27afaecf1a49/kube-rbac-proxy/0.log" Jul 09 12:04:33.870729 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:33.870699 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bjdxm_12901fb7-1801-4666-a17b-cc1d15663480/dns-node-resolver/0.log" Jul 09 12:04:34.870543 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:34.870516 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88pcx_153f22bc-d128-46ad-bb76-7ee70144b64b/node-ca/0.log" Jul 09 12:04:36.071239 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:04:36.071205 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qc2mn_9354067b-2812-4427-b351-ce4b19bb8b97/serve-healthcheck-canary/0.log" Jul 09 12:07:45.656980 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:07:45.656953 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 12:13:29.253315 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.253096 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm"] Jul 09 12:13:29.255441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.255419 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.259486 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.259466 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 12:13:29.259608 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.259497 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 12:13:29.259608 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.259499 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f8j4c\"" Jul 09 12:13:29.285271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.285237 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm"] Jul 09 12:13:29.378944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.378897 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-xc2rq"] Jul 09 12:13:29.380781 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.380768 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:13:29.380909 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.380891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:29.382619 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.382599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.386634 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.386609 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 12:13:29.386752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.386649 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2wxtv\"" Jul 09 12:13:29.386752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.386656 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 12:13:29.388948 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.388909 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dm4tp\"" Jul 09 12:13:29.388948 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.388921 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 12:13:29.389107 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.389016 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 12:13:29.389107 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.389029 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 12:13:29.389107 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.389070 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 12:13:29.390244 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.390228 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 12:13:29.393189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.393171 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 12:13:29.398703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.398683 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xc2rq"] Jul 09 12:13:29.406826 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.406803 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:13:29.427089 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.427070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.427196 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.427108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.457133 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.457107 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5"] Jul 09 12:13:29.459234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.459220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:29.464054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.464035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 12:13:29.464166 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.464104 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fjl89\"" Jul 09 12:13:29.466012 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.465994 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cccb88cf4-djfnb"] Jul 09 12:13:29.468110 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.468094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.471056 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.471040 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 12:13:29.471258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.471244 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r2nb5\"" Jul 09 12:13:29.471721 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.471708 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 12:13:29.471766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.471729 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 12:13:29.476913 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.476891 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5"] Jul 09 12:13:29.479187 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.479169 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 12:13:29.484513 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.484490 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cccb88cf4-djfnb"] Jul 09 12:13:29.528069 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.527994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528069 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.528069 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8n5f\" (UniqueName: \"kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528134 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srsjj\" (UniqueName: \"kubernetes.io/projected/9a5ee676-65ea-42ba-bc34-82449f7bbd4e-kube-api-access-srsjj\") pod \"downloads-685bfb5f64-xc2rq\" (UID: \"9a5ee676-65ea-42ba-bc34-82449f7bbd4e\") " pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:29.528272 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.528446 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.528831 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.528798 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.531174 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.531152 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ww6pm\" (UID: \"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.563550 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.563523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" Jul 09 12:13:29.629493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srsjj\" (UniqueName: \"kubernetes.io/projected/9a5ee676-65ea-42ba-bc34-82449f7bbd4e-kube-api-access-srsjj\") pod \"downloads-685bfb5f64-xc2rq\" (UID: \"9a5ee676-65ea-42ba-bc34-82449f7bbd4e\") " pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:29.629636 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d9f30c0-3f06-4644-92a5-6a09f74fe311-ca-trust-extracted\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.629636 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.629636 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-trusted-ca\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.629636 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwk4v\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-kube-api-access-gwk4v\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.629833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.629661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.630370 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630344 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-certificates\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.630479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-image-registry-private-configuration\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.630479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c22707e-a1f3-4c72-8d80-d1bb97d984a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-8crd5\" (UID: \"4c22707e-a1f3-4c72-8d80-d1bb97d984a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:29.630589 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-bound-sa-token\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.630589 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.630589 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8n5f\" (UniqueName: \"kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.630754 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.630600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.631356 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631274 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.631453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.631536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631512 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.631536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631403 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-tls\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.631632 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-installation-pull-secrets\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.631798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.631779 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.632520 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.632498 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.634155 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.633021 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.635739 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.635038 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.638626 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.638604 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.647045 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.647023 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8n5f\" (UniqueName: \"kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f\") pod \"console-84b775844d-mgnbj\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.647195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.647049 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srsjj\" (UniqueName: \"kubernetes.io/projected/9a5ee676-65ea-42ba-bc34-82449f7bbd4e-kube-api-access-srsjj\") pod \"downloads-685bfb5f64-xc2rq\" (UID: \"9a5ee676-65ea-42ba-bc34-82449f7bbd4e\") " pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:29.690788 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.690758 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:29.696560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.696544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:29.702296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.702274 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm"] Jul 09 12:13:29.705673 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:29.705642 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b134f5_0b4d_4b51_8bfe_12c8bfc7a3fa.slice/crio-683d23c2fc1f86bffb73f2c1d9da0b7ec6afff0cc531453ee7b04676cc4a5943 WatchSource:0}: Error finding container 683d23c2fc1f86bffb73f2c1d9da0b7ec6afff0cc531453ee7b04676cc4a5943: Status 404 returned error can't find the container with id 683d23c2fc1f86bffb73f2c1d9da0b7ec6afff0cc531453ee7b04676cc4a5943 Jul 09 12:13:29.707555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.707540 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.733883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-certificates\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.733955 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-image-registry-private-configuration\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.733988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c22707e-a1f3-4c72-8d80-d1bb97d984a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-8crd5\" (UID: \"4c22707e-a1f3-4c72-8d80-d1bb97d984a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-bound-sa-token\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734085 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-tls\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-installation-pull-secrets\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d9f30c0-3f06-4644-92a5-6a09f74fe311-ca-trust-extracted\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-trusted-ca\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.734812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.734219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwk4v\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-kube-api-access-gwk4v\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.735404 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.735318 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-certificates\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.735863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.735522 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d9f30c0-3f06-4644-92a5-6a09f74fe311-ca-trust-extracted\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.736304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.736280 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f30c0-3f06-4644-92a5-6a09f74fe311-trusted-ca\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.737224 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.737191 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-image-registry-private-configuration\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.737968 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.737946 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-registry-tls\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.738556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.738523 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c22707e-a1f3-4c72-8d80-d1bb97d984a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-8crd5\" (UID: \"4c22707e-a1f3-4c72-8d80-d1bb97d984a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:29.738784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.738765 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d9f30c0-3f06-4644-92a5-6a09f74fe311-installation-pull-secrets\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.745519 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.745487 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-bound-sa-token\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.746212 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.746178 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwk4v\" (UniqueName: \"kubernetes.io/projected/0d9f30c0-3f06-4644-92a5-6a09f74fe311-kube-api-access-gwk4v\") pod \"image-registry-cccb88cf4-djfnb\" (UID: \"0d9f30c0-3f06-4644-92a5-6a09f74fe311\") " pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.768836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.768580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:29.776826 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.776787 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:29.845104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.844696 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xc2rq"] Jul 09 12:13:29.849870 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:29.849843 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5ee676_65ea_42ba_bc34_82449f7bbd4e.slice/crio-85355b01548232fcd26e4201866b613b3033ed7f74a0b625cae24db410bbd5f2 WatchSource:0}: Error finding container 85355b01548232fcd26e4201866b613b3033ed7f74a0b625cae24db410bbd5f2: Status 404 returned error can't find the container with id 85355b01548232fcd26e4201866b613b3033ed7f74a0b625cae24db410bbd5f2 Jul 09 12:13:29.863202 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.863162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xc2rq" event={"ID":"9a5ee676-65ea-42ba-bc34-82449f7bbd4e","Type":"ContainerStarted","Data":"85355b01548232fcd26e4201866b613b3033ed7f74a0b625cae24db410bbd5f2"} Jul 09 12:13:29.865184 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.865132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" event={"ID":"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa","Type":"ContainerStarted","Data":"683d23c2fc1f86bffb73f2c1d9da0b7ec6afff0cc531453ee7b04676cc4a5943"} Jul 09 12:13:29.868844 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.868820 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:13:29.872000 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:29.871972 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12fcc0a_a345_4ad1_bdde_a7b20a5f2afc.slice/crio-d77a375a3caa7e7c6c4e1384190716d55a071da4a4199b03a7f33c3638b79a77 WatchSource:0}: Error finding container d77a375a3caa7e7c6c4e1384190716d55a071da4a4199b03a7f33c3638b79a77: Status 404 returned error can't find the container with id d77a375a3caa7e7c6c4e1384190716d55a071da4a4199b03a7f33c3638b79a77 Jul 09 12:13:29.912659 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.912628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5"] Jul 09 12:13:29.915772 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:29.915744 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c22707e_a1f3_4c72_8d80_d1bb97d984a8.slice/crio-d83b775701d02f8a0848d6d82a220f998dcf024590542da566eb502c06091d7e WatchSource:0}: Error finding container d83b775701d02f8a0848d6d82a220f998dcf024590542da566eb502c06091d7e: Status 404 returned error can't find the container with id d83b775701d02f8a0848d6d82a220f998dcf024590542da566eb502c06091d7e Jul 09 12:13:29.935026 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:29.935004 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cccb88cf4-djfnb"] Jul 09 12:13:29.937342 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:29.937313 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9f30c0_3f06_4644_92a5_6a09f74fe311.slice/crio-edf22f6dbb24109bf5aab3511a8ce33dd283973dd26c41562e1e53d7d6ebce0d WatchSource:0}: Error finding container edf22f6dbb24109bf5aab3511a8ce33dd283973dd26c41562e1e53d7d6ebce0d: Status 404 returned error can't find the container with id edf22f6dbb24109bf5aab3511a8ce33dd283973dd26c41562e1e53d7d6ebce0d Jul 09 12:13:30.872804 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.872767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b775844d-mgnbj" event={"ID":"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc","Type":"ContainerStarted","Data":"d77a375a3caa7e7c6c4e1384190716d55a071da4a4199b03a7f33c3638b79a77"} Jul 09 12:13:30.874820 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.874788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" event={"ID":"0d9f30c0-3f06-4644-92a5-6a09f74fe311","Type":"ContainerStarted","Data":"2447599935d1ced7d53a4556adcdd8051b93ddb570f4fe8f953c0f67640681db"} Jul 09 12:13:30.874969 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.874841 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" event={"ID":"0d9f30c0-3f06-4644-92a5-6a09f74fe311","Type":"ContainerStarted","Data":"edf22f6dbb24109bf5aab3511a8ce33dd283973dd26c41562e1e53d7d6ebce0d"} Jul 09 12:13:30.875595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.875572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:30.876944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.876904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" event={"ID":"4c22707e-a1f3-4c72-8d80-d1bb97d984a8","Type":"ContainerStarted","Data":"d83b775701d02f8a0848d6d82a220f998dcf024590542da566eb502c06091d7e"} Jul 09 12:13:30.902116 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:30.901398 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" podStartSLOduration=1.901379675 podStartE2EDuration="1.901379675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:30.899847009 +0000 UTC m=+645.662572566" watchObservedRunningTime="2026-07-09 12:13:30.901379675 +0000 UTC m=+645.664105210" Jul 09 12:13:31.883190 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.883150 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" event={"ID":"4c22707e-a1f3-4c72-8d80-d1bb97d984a8","Type":"ContainerStarted","Data":"4a001d4ab1bb3883764acde58546d38f34449982438aa14c669ae9e7372699ad"} Jul 09 12:13:31.883646 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.883364 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:31.885424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.885396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" event={"ID":"a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa","Type":"ContainerStarted","Data":"bfe9e44642b964b6c5700df38d47e719e524510e265e5142520715abb43877d8"} Jul 09 12:13:31.889536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.889514 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" Jul 09 12:13:31.899848 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.899798 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-8crd5" podStartSLOduration=1.410431803 podStartE2EDuration="2.899781027s" totalImagesPullingTime="1.489349224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:31.898792129 +0000 UTC m=+646.661517686" watchObservedRunningTime="2026-07-09 12:13:31.899781027 +0000 UTC m=+646.662506564" Jul 09 12:13:31.913388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:31.913158 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ww6pm" podStartSLOduration=1.216786325 podStartE2EDuration="2.913141138s" totalImagesPullingTime="1.696354813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:31.912888185 +0000 UTC m=+646.675613723" watchObservedRunningTime="2026-07-09 12:13:31.913141138 +0000 UTC m=+646.675866674" Jul 09 12:13:33.896524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:33.896480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b775844d-mgnbj" event={"ID":"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc","Type":"ContainerStarted","Data":"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb"} Jul 09 12:13:33.913953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:33.913869 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b775844d-mgnbj" podStartSLOduration=1.4014412680000001 podStartE2EDuration="4.913848983s" totalImagesPullingTime="3.512407715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:33.913189059 +0000 UTC m=+648.675914597" watchObservedRunningTime="2026-07-09 12:13:33.913848983 +0000 UTC m=+648.676574521" Jul 09 12:13:38.254296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.254263 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bnm9v"] Jul 09 12:13:38.256986 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.256958 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.260427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.260402 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 12:13:38.260702 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.260588 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 12:13:38.260702 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.260626 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 12:13:38.261427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.261407 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 12:13:38.261645 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.261626 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 12:13:38.262717 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.262699 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5nslp\"" Jul 09 12:13:38.268637 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.265823 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 12:13:38.300208 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-root\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300334 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300237 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-sys\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300334 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-textfile\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-tls\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300384 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300424 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8wn\" (UniqueName: \"kubernetes.io/projected/2f7180d2-2f04-453e-bb5a-02693c15d8b4-kube-api-access-tf8wn\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300439 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-wtmp\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-metrics-client-ca\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.300549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.300486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.401258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-sys\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.401258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-textfile\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.401258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-tls\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.401258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401242 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-sys\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8wn\" (UniqueName: \"kubernetes.io/projected/2f7180d2-2f04-453e-bb5a-02693c15d8b4-kube-api-access-tf8wn\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-wtmp\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-metrics-client-ca\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-root\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401499 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-root\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401582 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-textfile\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.401954 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.402078 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7180d2-2f04-453e-bb5a-02693c15d8b4-metrics-client-ca\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.402213 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-wtmp\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.404689 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-tls\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.404761 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f7180d2-2f04-453e-bb5a-02693c15d8b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.416219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.410048 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8wn\" (UniqueName: \"kubernetes.io/projected/2f7180d2-2f04-453e-bb5a-02693c15d8b4-kube-api-access-tf8wn\") pod \"node-exporter-bnm9v\" (UID: \"2f7180d2-2f04-453e-bb5a-02693c15d8b4\") " pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.571229 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.571143 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bnm9v" Jul 09 12:13:38.583967 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:38.583918 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7180d2_2f04_453e_bb5a_02693c15d8b4.slice/crio-46063df8ed31ac87337cb9e5ee38ecb7ec0b6a3b6430fb122a38f9641879cb18 WatchSource:0}: Error finding container 46063df8ed31ac87337cb9e5ee38ecb7ec0b6a3b6430fb122a38f9641879cb18: Status 404 returned error can't find the container with id 46063df8ed31ac87337cb9e5ee38ecb7ec0b6a3b6430fb122a38f9641879cb18 Jul 09 12:13:38.918052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:38.917967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9v" event={"ID":"2f7180d2-2f04-453e-bb5a-02693c15d8b4","Type":"ContainerStarted","Data":"46063df8ed31ac87337cb9e5ee38ecb7ec0b6a3b6430fb122a38f9641879cb18"} Jul 09 12:13:39.309104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.309067 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 12:13:39.312336 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.312309 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.315234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315206 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 12:13:39.315234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315228 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 12:13:39.315417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315252 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 12:13:39.315417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315298 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m76xq\"" Jul 09 12:13:39.315417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315234 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 12:13:39.315417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315210 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 12:13:39.315603 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315550 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 12:13:39.315769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315702 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 12:13:39.315863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315829 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 12:13:39.315863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.315853 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 12:13:39.330427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.330341 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-config-out\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-web-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409669 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409919 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpp2\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-kube-api-access-klpp2\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.410118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.409986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.510762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.510762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.510762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpp2\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-kube-api-access-klpp2\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510872 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510899 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-config-out\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-web-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.510982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.511005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.511030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.511061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511545 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.511123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.511545 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:13:39.511504 2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 09 12:13:39.511640 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:13:39.511575 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls podName:6e7aac14-b869-4c50-8336-a74605822a30 nodeName:}" failed. No retries permitted until 2026-07-09 12:13:40.011554647 +0000 UTC m=+654.774280178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6e7aac14-b869-4c50-8336-a74605822a30") : secret "alertmanager-main-tls" not found Jul 09 12:13:39.511833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.511802 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.512321 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.512148 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.513226 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.513170 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e7aac14-b869-4c50-8336-a74605822a30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.515217 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.515175 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e7aac14-b869-4c50-8336-a74605822a30-config-out\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.515598 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.515576 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-web-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.515675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.515606 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.516031 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.515986 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.516111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.516079 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.516165 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.516109 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.516523 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.516501 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.517493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.517467 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.523129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.523106 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpp2\" (UniqueName: \"kubernetes.io/projected/6e7aac14-b869-4c50-8336-a74605822a30-kube-api-access-klpp2\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:39.696916 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.696876 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:39.697280 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.697253 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:39.702994 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.702949 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:39.923382 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.923354 2567 generic.go:356] "Generic (PLEG): container finished" podID="2f7180d2-2f04-453e-bb5a-02693c15d8b4" containerID="e382dc97a007d4e7b63d18578f53d4d760f44bad5ec04a0501c5b09c3fe36cb6" exitCode=0 Jul 09 12:13:39.923515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.923432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9v" event={"ID":"2f7180d2-2f04-453e-bb5a-02693c15d8b4","Type":"ContainerDied","Data":"e382dc97a007d4e7b63d18578f53d4d760f44bad5ec04a0501c5b09c3fe36cb6"} Jul 09 12:13:39.928070 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:39.928051 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:13:40.014534 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.014474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:40.017772 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.017745 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e7aac14-b869-4c50-8336-a74605822a30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e7aac14-b869-4c50-8336-a74605822a30\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:40.224868 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.224831 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 12:13:40.380671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.380636 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 12:13:40.384610 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:40.384581 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7aac14_b869_4c50_8336_a74605822a30.slice/crio-5ade97a2055be0198828905a7cff6b7cf5a71f123cee8b8fcfbab4b8df86465c WatchSource:0}: Error finding container 5ade97a2055be0198828905a7cff6b7cf5a71f123cee8b8fcfbab4b8df86465c: Status 404 returned error can't find the container with id 5ade97a2055be0198828905a7cff6b7cf5a71f123cee8b8fcfbab4b8df86465c Jul 09 12:13:40.929005 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.928962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9v" event={"ID":"2f7180d2-2f04-453e-bb5a-02693c15d8b4","Type":"ContainerStarted","Data":"787bb1a72318c2531565f196003d7721e83d7174c2700ea29b22ca8ad21bf005"} Jul 09 12:13:40.929196 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.929037 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9v" event={"ID":"2f7180d2-2f04-453e-bb5a-02693c15d8b4","Type":"ContainerStarted","Data":"d4e0edfc72c94f1be3e69d976e39d9603362e88c202d6e85c7a041edffc38286"} Jul 09 12:13:40.930577 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.930554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"5ade97a2055be0198828905a7cff6b7cf5a71f123cee8b8fcfbab4b8df86465c"} Jul 09 12:13:40.951667 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:40.951624 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bnm9v" podStartSLOduration=1.69273751 podStartE2EDuration="2.951603149s" totalImagesPullingTime="1.180365098s" totalInitContainerRuntime="78.500541ms" isStatefulPod=true podCreationTimestamp="2026-07-09 12:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:40.951054128 +0000 UTC m=+655.713779664" watchObservedRunningTime="2026-07-09 12:13:40.951603149 +0000 UTC m=+655.714328677" Jul 09 12:13:41.221199 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.220602 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j"] Jul 09 12:13:41.224279 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.223796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.226452 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.226424 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-efbv6ukfrr2io\"" Jul 09 12:13:41.226671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.226654 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dx5fd\"" Jul 09 12:13:41.227080 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.226917 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 12:13:41.227080 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.226920 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 12:13:41.227080 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.226973 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 12:13:41.227292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.227222 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 12:13:41.227292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.227243 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 12:13:41.236900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.236822 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j"] Jul 09 12:13:41.324453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324465 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-metrics-client-ca\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324606 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crq7\" (UniqueName: \"kubernetes.io/projected/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-kube-api-access-9crq7\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324837 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324677 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324837 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.324837 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.324767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-grpc-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-metrics-client-ca\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crq7\" (UniqueName: \"kubernetes.io/projected/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-kube-api-access-9crq7\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.426314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-grpc-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.427836 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-metrics-client-ca\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.430523 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.431468 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.431110 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.434057 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.433305 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.436663 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.436619 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.437135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.437114 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crq7\" (UniqueName: \"kubernetes.io/projected/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-kube-api-access-9crq7\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.437986 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.437964 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-grpc-tls\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.438266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.438222 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6eedb07d-bb05-4f16-8c5d-0e6ef904568a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759bd7bdc4-9wt6j\" (UID: \"6eedb07d-bb05-4f16-8c5d-0e6ef904568a\") " pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.537756 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.537650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:41.985201 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:41.985045 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j"] Jul 09 12:13:42.019187 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:42.019153 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eedb07d_bb05_4f16_8c5d_0e6ef904568a.slice/crio-270c45b24b02f544a1dc862f40e8f859a0775ce13ec2b8a611c40b463c4a0590 WatchSource:0}: Error finding container 270c45b24b02f544a1dc862f40e8f859a0775ce13ec2b8a611c40b463c4a0590: Status 404 returned error can't find the container with id 270c45b24b02f544a1dc862f40e8f859a0775ce13ec2b8a611c40b463c4a0590 Jul 09 12:13:42.941881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:42.941850 2567 generic.go:356] "Generic (PLEG): container finished" podID="6e7aac14-b869-4c50-8336-a74605822a30" containerID="f33855d23a944974c0fc6c02b268faabb9565b5d08f2b6f80ea2dbf222f818ee" exitCode=0 Jul 09 12:13:42.942364 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:42.942266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerDied","Data":"f33855d23a944974c0fc6c02b268faabb9565b5d08f2b6f80ea2dbf222f818ee"} Jul 09 12:13:42.944458 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:42.944412 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"270c45b24b02f544a1dc862f40e8f859a0775ce13ec2b8a611c40b463c4a0590"} Jul 09 12:13:43.007997 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.007962 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7"] Jul 09 12:13:43.015119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.014876 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:13:43.015401 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.015372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.017467 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.017440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.019083 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.019058 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6dg75\"" Jul 09 12:13:43.019217 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.019190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 12:13:43.024322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.024301 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7"] Jul 09 12:13:43.039054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.039031 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:13:43.040699 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.040806 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.040806 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040755 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq5wt\" (UniqueName: \"kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.040806 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040783 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.041046 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.041046 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.040964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.041368 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.041320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmws\" (UniqueName: \"kubernetes.io/projected/97be61b3-733b-47de-ab1c-483d79194fe1-kube-api-access-hwmws\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.042570 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.041642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.042570 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.041881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmws\" (UniqueName: \"kubernetes.io/projected/97be61b3-733b-47de-ab1c-483d79194fe1-kube-api-access-hwmws\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.143676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143843 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143843 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143721 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq5wt\" (UniqueName: \"kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143843 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.143843 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.143815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.144087 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:13:43.143964 2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 09 12:13:43.144087 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:13:43.144023 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert podName:97be61b3-733b-47de-ab1c-483d79194fe1 nodeName:}" failed. No retries permitted until 2026-07-09 12:13:43.644004562 +0000 UTC m=+658.406730081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-f7cb7" (UID: "97be61b3-733b-47de-ab1c-483d79194fe1") : secret "monitoring-plugin-cert" not found Jul 09 12:13:43.144349 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.144324 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.144444 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.144419 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.144653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.144613 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.145083 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.145053 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.146583 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.146554 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.146909 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.146888 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.152650 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.152601 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmws\" (UniqueName: \"kubernetes.io/projected/97be61b3-733b-47de-ab1c-483d79194fe1-kube-api-access-hwmws\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.152914 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.152810 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq5wt\" (UniqueName: \"kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt\") pod \"console-77dd4895c7-b2fx7\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.334799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.334762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:43.648318 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.648285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.651121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.651095 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97be61b3-733b-47de-ab1c-483d79194fe1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-f7cb7\" (UID: \"97be61b3-733b-47de-ab1c-483d79194fe1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:43.929276 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:43.929201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:49.956048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.956018 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:13:49.959623 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:49.959586 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec99ccd_9db1_4f7b_8eda_85fe3b73ed2f.slice/crio-5d684fab0fadb55cb34f27227484d7920ac583c871f562def1db2635bbb0bd93 WatchSource:0}: Error finding container 5d684fab0fadb55cb34f27227484d7920ac583c871f562def1db2635bbb0bd93: Status 404 returned error can't find the container with id 5d684fab0fadb55cb34f27227484d7920ac583c871f562def1db2635bbb0bd93 Jul 09 12:13:49.971884 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.971851 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7"] Jul 09 12:13:49.972403 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.972346 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dd4895c7-b2fx7" event={"ID":"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f","Type":"ContainerStarted","Data":"5d684fab0fadb55cb34f27227484d7920ac583c871f562def1db2635bbb0bd93"} Jul 09 12:13:49.974002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.973965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xc2rq" event={"ID":"9a5ee676-65ea-42ba-bc34-82449f7bbd4e","Type":"ContainerStarted","Data":"60ff224e35bab1750d6ea0ee75c231d0ff79d863f4ffc472483aa0931afd6038"} Jul 09 12:13:49.974354 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.974331 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:49.975550 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.975524 2567 patch_prober.go:28] interesting pod/downloads-685bfb5f64-xc2rq container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body= Jul 09 12:13:49.975643 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.975576 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-xc2rq" podUID="9a5ee676-65ea-42ba-bc34-82449f7bbd4e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" Jul 09 12:13:49.975766 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:13:49.975711 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97be61b3_733b_47de_ab1c_483d79194fe1.slice/crio-2c064ef82baaba566bc4ee8dfa63979d9ba20c8fc2cbac4fae257043ab1e53c7 WatchSource:0}: Error finding container 2c064ef82baaba566bc4ee8dfa63979d9ba20c8fc2cbac4fae257043ab1e53c7: Status 404 returned error can't find the container with id 2c064ef82baaba566bc4ee8dfa63979d9ba20c8fc2cbac4fae257043ab1e53c7 Jul 09 12:13:49.992867 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:49.992642 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-xc2rq" podStartSLOduration=0.945561274 podStartE2EDuration="20.992623687s" totalImagesPullingTime="20.047062413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:49.992427743 +0000 UTC m=+664.755153292" watchObservedRunningTime="2026-07-09 12:13:49.992623687 +0000 UTC m=+664.755349224" Jul 09 12:13:50.981961 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:50.981310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dd4895c7-b2fx7" event={"ID":"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f","Type":"ContainerStarted","Data":"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97"} Jul 09 12:13:50.985771 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:50.985707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" event={"ID":"97be61b3-733b-47de-ab1c-483d79194fe1","Type":"ContainerStarted","Data":"2c064ef82baaba566bc4ee8dfa63979d9ba20c8fc2cbac4fae257043ab1e53c7"} Jul 09 12:13:50.996998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:50.996904 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-xc2rq" Jul 09 12:13:51.007028 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:51.006742 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77dd4895c7-b2fx7" podStartSLOduration=9.006727303 podStartE2EDuration="9.006727303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:51.004130426 +0000 UTC m=+665.766855985" watchObservedRunningTime="2026-07-09 12:13:51.006727303 +0000 UTC m=+665.769452839" Jul 09 12:13:52.897200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:52.897165 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cccb88cf4-djfnb" Jul 09 12:13:53.335075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:53.335039 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:53.335275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:53.335107 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:53.340489 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:53.340462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:54.002595 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.002452 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" event={"ID":"97be61b3-733b-47de-ab1c-483d79194fe1","Type":"ContainerStarted","Data":"8ed819905a7b35ac9df10cc2c7cf8d16a519c788d5d07adb92faa4395b426b35"} Jul 09 12:13:54.003627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.003606 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:54.007419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.007163 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"d933b4a3cadfe2b7a4b4d8cfa0f29c127f85cebe44b603a7837eea725e0c4ace"} Jul 09 12:13:54.007419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.007264 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"20fcbeaa07d0ebce93aff540d78d910e84c3e6b96c649d3ad0d2949bf764f79c"} Jul 09 12:13:54.010752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.010374 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"8bc77e20a83efa2bd4543754936c00bbf7721fb4ea32439f40081d3fc9076918"} Jul 09 12:13:54.010752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.010541 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"7ce77555b17d0f8cd6aff75a20b83e49e106ebe836c8f3a1af8098690cf419e0"} Jul 09 12:13:54.015991 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.015954 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" Jul 09 12:13:54.016574 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.016555 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:13:54.032815 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.032773 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-f7cb7" podStartSLOduration=8.317360019 podStartE2EDuration="12.032759549s" totalImagesPullingTime="3.71539953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:54.031534533 +0000 UTC m=+668.794260091" watchObservedRunningTime="2026-07-09 12:13:54.032759549 +0000 UTC m=+668.795485108" Jul 09 12:13:54.094634 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:54.094009 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:13:55.019535 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:55.019483 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"32188366eecff6343659a0d07828fa57b6235599bebb13dff5a98231dbd3e3eb"} Jul 09 12:13:55.020091 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:55.019633 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"be4deed83cd74af3f5ab2f3540c42a6a60ad5476ab6742c5335e47989395fb94"} Jul 09 12:13:55.020091 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:55.019676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"198f157575a2fe74e76065157646406b119ad56273698d98b36887d9ca3fe6a8"} Jul 09 12:13:55.023280 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:55.023166 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"5fb1d6471cefc0c835fa8ae4352b499a5f2eb0c4d27858adf5754cc8a8577bca"} Jul 09 12:13:56.033281 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:56.033241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e7aac14-b869-4c50-8336-a74605822a30","Type":"ContainerStarted","Data":"c72732e6c6df4c4befeac1ec51b4e23c70823b30be2b30bbe16669e5fda4fb61"} Jul 09 12:13:56.036721 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:56.036668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"ec62cfb07d8328bf90c904ab858ee407d036925965b0c91d1bacd46eb9fc68f3"} Jul 09 12:13:56.060737 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:56.060677 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.410219841 podStartE2EDuration="17.060655155s" totalImagesPullingTime="13.633309736s" totalInitContainerRuntime="17.125578ms" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:56.058907433 +0000 UTC m=+670.821632995" watchObservedRunningTime="2026-07-09 12:13:56.060655155 +0000 UTC m=+670.823380695" Jul 09 12:13:57.046179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:57.046134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"13dd0645e9518cf684197b7c011a63a4c7adb84a28333ec453d0b7d3318cab36"} Jul 09 12:13:57.046709 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:57.046255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" event={"ID":"6eedb07d-bb05-4f16-8c5d-0e6ef904568a","Type":"ContainerStarted","Data":"e54133590e0b97d0aa4732266d34c2ad07a82a0f2746f7a0660c98fea1e0a183"} Jul 09 12:13:57.046709 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:57.046664 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:13:57.074740 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:57.074680 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" podStartSLOduration=2.726174706 podStartE2EDuration="16.074662178s" totalImagesPullingTime="13.348487472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:13:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:13:57.074649485 +0000 UTC m=+671.837375238" watchObservedRunningTime="2026-07-09 12:13:57.074662178 +0000 UTC m=+671.837387711" Jul 09 12:13:58.057798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:13:58.057764 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-759bd7bdc4-9wt6j" Jul 09 12:14:19.124470 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.124403 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84b775844d-mgnbj" podUID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" containerName="console" containerID="cri-o://e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb" gracePeriod=15 Jul 09 12:14:19.397427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.397372 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:14:19.569801 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.569768 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570013 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.569817 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570013 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.569839 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570013 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.569869 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570080 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8n5f\" (UniqueName: \"kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570136 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert\") pod \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\" (UID: \"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc\") " Jul 09 12:14:19.570318 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570261 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca" (OuterVolumeSpecName: "service-ca") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:14:19.570318 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570267 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config" (OuterVolumeSpecName: "console-config") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:14:19.570430 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570410 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-service-ca\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.570560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570545 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.570631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570406 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:14:19.570631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.570588 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:14:19.572400 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.572367 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:14:19.572400 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.572382 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:14:19.572544 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.572412 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f" (OuterVolumeSpecName: "kube-api-access-j8n5f") pod "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" (UID: "c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc"). InnerVolumeSpecName "kube-api-access-j8n5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:14:19.671761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.671697 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-trusted-ca-bundle\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.671761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.671719 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-oauth-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.671761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.671729 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-oauth-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.671761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.671738 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8n5f\" (UniqueName: \"kubernetes.io/projected/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-kube-api-access-j8n5f\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:19.671761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:19.671746 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc-console-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:14:20.135116 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.135083 2567 generic.go:356] "Generic (PLEG): container finished" podID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" containerID="e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb" exitCode=0 Jul 09 12:14:20.135498 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.135129 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b775844d-mgnbj" Jul 09 12:14:20.135498 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.135166 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b775844d-mgnbj" event={"ID":"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc","Type":"ContainerDied","Data":"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb"} Jul 09 12:14:20.135498 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.135215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b775844d-mgnbj" event={"ID":"c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc","Type":"ContainerDied","Data":"d77a375a3caa7e7c6c4e1384190716d55a071da4a4199b03a7f33c3638b79a77"} Jul 09 12:14:20.135498 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.135246 2567 scope.go:117] "RemoveContainer" containerID="e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb" Jul 09 12:14:20.143541 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.143525 2567 scope.go:117] "RemoveContainer" containerID="e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb" Jul 09 12:14:20.143790 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:14:20.143763 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb\": container with ID starting with e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb not found: ID does not exist" containerID="e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb" Jul 09 12:14:20.143848 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.143792 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb"} err="failed to get container status \"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb\": rpc error: code = NotFound desc = could not find container \"e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb\": container with ID starting with e21c02fdb2648b7d95d061841be8c22c87e711c1d8133766a922c67f546a3edb not found: ID does not exist" Jul 09 12:14:20.155295 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.152712 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:14:20.156689 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:20.156666 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b775844d-mgnbj"] Jul 09 12:14:21.846449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:14:21.846413 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" path="/var/lib/kubelet/pods/c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc/volumes" Jul 09 12:15:02.620994 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.620954 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c97df5865-2jzfc"] Jul 09 12:15:02.621449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.621264 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" containerName="console" Jul 09 12:15:02.621449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.621274 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" containerName="console" Jul 09 12:15:02.621449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.621333 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12fcc0a-a345-4ad1-bdde-a7b20a5f2afc" containerName="console" Jul 09 12:15:02.624180 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.624163 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.626653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.626633 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 12:15:02.628368 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.628351 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 12:15:02.628461 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.628424 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-65sfd\"" Jul 09 12:15:02.628580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.628565 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 12:15:02.628668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.628656 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 12:15:02.629369 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.629353 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 12:15:02.634228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.634206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 12:15:02.652452 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.652421 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c97df5865-2jzfc"] Jul 09 12:15:02.708694 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.708887 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslgr\" (UniqueName: \"kubernetes.io/projected/0202d18d-d511-4820-a351-5e8a37a2cb1f-kube-api-access-xslgr\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.708887 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-serving-certs-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.708887 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.708887 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.708887 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-metrics-client-ca\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.709111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.709111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.708998 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-federate-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.809953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.809890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.809965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslgr\" (UniqueName: \"kubernetes.io/projected/0202d18d-d511-4820-a351-5e8a37a2cb1f-kube-api-access-xslgr\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810001 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-serving-certs-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-metrics-client-ca\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810185 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-federate-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.810949 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.810899 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-metrics-client-ca\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.811155 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.811135 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-serving-certs-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.811332 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.811306 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.812575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.812545 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.812881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.812864 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-federate-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.813205 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.813182 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-telemeter-client-tls\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.813271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.813186 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0202d18d-d511-4820-a351-5e8a37a2cb1f-secret-telemeter-client\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.819191 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.819169 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslgr\" (UniqueName: \"kubernetes.io/projected/0202d18d-d511-4820-a351-5e8a37a2cb1f-kube-api-access-xslgr\") pod \"telemeter-client-c97df5865-2jzfc\" (UID: \"0202d18d-d511-4820-a351-5e8a37a2cb1f\") " pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:02.933859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:02.933768 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" Jul 09 12:15:03.103548 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:03.103502 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c97df5865-2jzfc"] Jul 09 12:15:03.105725 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:15:03.105697 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0202d18d_d511_4820_a351_5e8a37a2cb1f.slice/crio-da2085e4436f29b2227664ae1045d60f879e787699b9ccee0b1d8a29e9109070 WatchSource:0}: Error finding container da2085e4436f29b2227664ae1045d60f879e787699b9ccee0b1d8a29e9109070: Status 404 returned error can't find the container with id da2085e4436f29b2227664ae1045d60f879e787699b9ccee0b1d8a29e9109070 Jul 09 12:15:03.296104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:03.296072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" event={"ID":"0202d18d-d511-4820-a351-5e8a37a2cb1f","Type":"ContainerStarted","Data":"da2085e4436f29b2227664ae1045d60f879e787699b9ccee0b1d8a29e9109070"} Jul 09 12:15:06.309246 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:06.309211 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" event={"ID":"0202d18d-d511-4820-a351-5e8a37a2cb1f","Type":"ContainerStarted","Data":"f44d00e11d2a5cdd8e067f4f0cac05de8c23d7a8471219f37f5b8524b558c3c7"} Jul 09 12:15:06.309708 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:06.309270 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" event={"ID":"0202d18d-d511-4820-a351-5e8a37a2cb1f","Type":"ContainerStarted","Data":"a1fe862e4893a486e18567891e3f3821481289f44c1c9b4d8c6b9738ef4f0181"} Jul 09 12:15:06.309708 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:06.309291 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" event={"ID":"0202d18d-d511-4820-a351-5e8a37a2cb1f","Type":"ContainerStarted","Data":"4cab9e021b72aaed6ee9ac5b20b6d45a15894f9dff8f65623b2ec15c3b8528c8"} Jul 09 12:15:06.336293 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:06.336182 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c97df5865-2jzfc" podStartSLOduration=1.364045559 podStartE2EDuration="4.336166945s" totalImagesPullingTime="2.972121386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:15:06.334884801 +0000 UTC m=+741.097610345" watchObservedRunningTime="2026-07-09 12:15:06.336166945 +0000 UTC m=+741.098892481" Jul 09 12:15:07.217052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.217022 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:07.219453 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.219432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.233052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.233026 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:07.353462 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353462 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353467 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7k6\" (UniqueName: \"kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.353883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.353666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454106 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454068 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454106 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7k6\" (UniqueName: \"kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.454998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454966 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.455135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.454976 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.455135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.455031 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.455968 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.455917 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.456953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.456898 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.457367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.457348 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.465427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.465402 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7k6\" (UniqueName: \"kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6\") pod \"console-55658487d4-h5vcf\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.528193 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.528101 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:07.662706 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:07.662676 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:07.665463 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:15:07.665433 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3ea18_dd8e_40b1_b6f7_2b5bd74fe7ea.slice/crio-29fcb0112cec90b8ced8fa6337ad00371b5ea26445e9dfb4b2d9216b438101c9 WatchSource:0}: Error finding container 29fcb0112cec90b8ced8fa6337ad00371b5ea26445e9dfb4b2d9216b438101c9: Status 404 returned error can't find the container with id 29fcb0112cec90b8ced8fa6337ad00371b5ea26445e9dfb4b2d9216b438101c9 Jul 09 12:15:08.318450 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:08.318416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55658487d4-h5vcf" event={"ID":"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea","Type":"ContainerStarted","Data":"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7"} Jul 09 12:15:08.318638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:08.318478 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55658487d4-h5vcf" event={"ID":"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea","Type":"ContainerStarted","Data":"29fcb0112cec90b8ced8fa6337ad00371b5ea26445e9dfb4b2d9216b438101c9"} Jul 09 12:15:17.528230 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:17.528189 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:17.528721 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:17.528248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:17.533039 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:17.533016 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:17.554777 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:17.554729 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55658487d4-h5vcf" podStartSLOduration=10.554714693 podStartE2EDuration="10.554714693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:15:08.33830804 +0000 UTC m=+743.101033575" watchObservedRunningTime="2026-07-09 12:15:17.554714693 +0000 UTC m=+752.317440221" Jul 09 12:15:18.365866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.365831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:18.417173 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.417141 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:15:18.557145 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.557113 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:15:18.559484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.559467 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.572797 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.572773 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:15:18.656972 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.656865 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.656972 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.656907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.656972 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.656951 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslz7\" (UniqueName: \"kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.657200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.656990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.657200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.657020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.657200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.657044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.657200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.657076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslz7\" (UniqueName: \"kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758278 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.758638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.758402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.759195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.759123 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.759195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.759148 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.759195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.759217 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.759500 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.759211 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.761024 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.760995 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.761158 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.761139 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.767478 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.767457 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslz7\" (UniqueName: \"kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7\") pod \"console-5d995c7998-spnpc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:18.868682 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:18.868642 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:19.004043 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:19.004020 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:15:19.006603 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:15:19.006574 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a847330_acf4_41c3_9909_04bd8b6021cc.slice/crio-f633d7ebcf2c699074226961991f34e3c9da87c7846c72247aef1d8dfcc26c9f WatchSource:0}: Error finding container f633d7ebcf2c699074226961991f34e3c9da87c7846c72247aef1d8dfcc26c9f: Status 404 returned error can't find the container with id f633d7ebcf2c699074226961991f34e3c9da87c7846c72247aef1d8dfcc26c9f Jul 09 12:15:19.366893 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:19.366857 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d995c7998-spnpc" event={"ID":"1a847330-acf4-41c3-9909-04bd8b6021cc","Type":"ContainerStarted","Data":"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9"} Jul 09 12:15:19.367102 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:19.366904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d995c7998-spnpc" event={"ID":"1a847330-acf4-41c3-9909-04bd8b6021cc","Type":"ContainerStarted","Data":"f633d7ebcf2c699074226961991f34e3c9da87c7846c72247aef1d8dfcc26c9f"} Jul 09 12:15:19.389800 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:19.389751 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d995c7998-spnpc" podStartSLOduration=1.389735663 podStartE2EDuration="1.389735663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:15:19.388198337 +0000 UTC m=+754.150923894" watchObservedRunningTime="2026-07-09 12:15:19.389735663 +0000 UTC m=+754.152461199" Jul 09 12:15:28.869254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:28.869216 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:28.869684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:28.869269 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:28.874055 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:28.874035 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:29.408087 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:29.408055 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:15:29.456671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:29.456643 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:43.439911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.439844 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-77dd4895c7-b2fx7" podUID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" containerName="console" containerID="cri-o://8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97" gracePeriod=15 Jul 09 12:15:43.673893 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.673870 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:15:43.773203 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773116 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773203 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773151 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773203 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773180 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773227 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq5wt\" (UniqueName: \"kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773289 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773451 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config\") pod \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\" (UID: \"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f\") " Jul 09 12:15:43.773675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773631 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:43.773675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773638 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:43.773769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773705 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:43.773821 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773777 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-trusted-ca-bundle\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.773821 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773795 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-oauth-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.773821 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773810 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-service-ca\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.774013 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.773993 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config" (OuterVolumeSpecName: "console-config") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:43.775648 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.775626 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:15:43.776067 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.776048 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:15:43.776170 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.776082 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt" (OuterVolumeSpecName: "kube-api-access-mq5wt") pod "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" (UID: "8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f"). InnerVolumeSpecName "kube-api-access-mq5wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:15:43.875230 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.875201 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-oauth-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.875230 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.875224 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq5wt\" (UniqueName: \"kubernetes.io/projected/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-kube-api-access-mq5wt\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.875395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.875234 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:43.875395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:43.875246 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f-console-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:44.465074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.465036 2567 generic.go:356] "Generic (PLEG): container finished" podID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" containerID="8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97" exitCode=0 Jul 09 12:15:44.465536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.465082 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dd4895c7-b2fx7" Jul 09 12:15:44.465536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.465111 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dd4895c7-b2fx7" event={"ID":"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f","Type":"ContainerDied","Data":"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97"} Jul 09 12:15:44.465536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.465162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dd4895c7-b2fx7" event={"ID":"8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f","Type":"ContainerDied","Data":"5d684fab0fadb55cb34f27227484d7920ac583c871f562def1db2635bbb0bd93"} Jul 09 12:15:44.465536 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.465184 2567 scope.go:117] "RemoveContainer" containerID="8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97" Jul 09 12:15:44.475545 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.475518 2567 scope.go:117] "RemoveContainer" containerID="8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97" Jul 09 12:15:44.475799 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:15:44.475778 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97\": container with ID starting with 8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97 not found: ID does not exist" containerID="8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97" Jul 09 12:15:44.475869 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.475807 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97"} err="failed to get container status \"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97\": rpc error: code = NotFound desc = could not find container \"8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97\": container with ID starting with 8c9c32817cfdf8bc8b73793df37b2cc313bed6ea4932cc55d2d3b4479f2e8e97 not found: ID does not exist" Jul 09 12:15:44.483563 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.483540 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:15:44.487490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:44.487471 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77dd4895c7-b2fx7"] Jul 09 12:15:45.846640 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:45.846605 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" path="/var/lib/kubelet/pods/8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f/volumes" Jul 09 12:15:54.477182 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.477116 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-55658487d4-h5vcf" podUID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" containerName="console" containerID="cri-o://335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7" gracePeriod=15 Jul 09 12:15:54.727518 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.727460 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:54.860147 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860113 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860172 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860198 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860215 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860291 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860565 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860341 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb7k6\" (UniqueName: \"kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6\") pod \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\" (UID: \"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea\") " Jul 09 12:15:54.860671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860644 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:54.860734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860673 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:54.860734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860709 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config" (OuterVolumeSpecName: "console-config") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:54.860814 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.860739 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:15:54.862677 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.862656 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6" (OuterVolumeSpecName: "kube-api-access-jb7k6") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "kube-api-access-jb7k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:15:54.862976 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.862956 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:15:54.863051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.862975 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" (UID: "aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:15:54.961615 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961581 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961615 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961608 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-trusted-ca-bundle\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961615 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961618 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-service-ca\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961628 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb7k6\" (UniqueName: \"kubernetes.io/projected/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-kube-api-access-jb7k6\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961638 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-oauth-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961646 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-console-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:54.961833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:54.961655 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea-oauth-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:15:55.506607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.506571 2567 generic.go:356] "Generic (PLEG): container finished" podID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" containerID="335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7" exitCode=0 Jul 09 12:15:55.507030 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.506617 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55658487d4-h5vcf" Jul 09 12:15:55.507030 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.506666 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55658487d4-h5vcf" event={"ID":"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea","Type":"ContainerDied","Data":"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7"} Jul 09 12:15:55.507030 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.506710 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55658487d4-h5vcf" event={"ID":"aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea","Type":"ContainerDied","Data":"29fcb0112cec90b8ced8fa6337ad00371b5ea26445e9dfb4b2d9216b438101c9"} Jul 09 12:15:55.507030 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.506731 2567 scope.go:117] "RemoveContainer" containerID="335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7" Jul 09 12:15:55.515872 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.515855 2567 scope.go:117] "RemoveContainer" containerID="335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7" Jul 09 12:15:55.516138 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:15:55.516120 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7\": container with ID starting with 335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7 not found: ID does not exist" containerID="335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7" Jul 09 12:15:55.516186 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.516147 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7"} err="failed to get container status \"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7\": rpc error: code = NotFound desc = could not find container \"335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7\": container with ID starting with 335a0842797620cf4b833fb350af63dde2bd54c5a0bb2184ba46b2796089b7f7 not found: ID does not exist" Jul 09 12:15:55.526966 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.526921 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:55.535794 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.535776 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55658487d4-h5vcf"] Jul 09 12:15:55.846337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:15:55.846261 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" path="/var/lib/kubelet/pods/aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea/volumes" Jul 09 12:16:23.705693 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.705617 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8"] Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.705923 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" containerName="console" Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.705950 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" containerName="console" Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.705964 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" containerName="console" Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.705969 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" containerName="console" Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.706032 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec99ccd-9db1-4f7b-8eda-85fe3b73ed2f" containerName="console" Jul 09 12:16:23.706118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.706044 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba3ea18-dd8e-40b1-b6f7-2b5bd74fe7ea" containerName="console" Jul 09 12:16:23.708938 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.708915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.711422 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.711390 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 12:16:23.711572 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.711402 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 12:16:23.712384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.712352 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 12:16:23.712384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.712365 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 12:16:23.712548 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.712352 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 12:16:23.712548 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.712428 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 12:16:23.712548 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.712444 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 12:16:23.721382 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.721361 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8"] Jul 09 12:16:23.803953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.803892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjtt\" (UniqueName: \"kubernetes.io/projected/1d83067e-4a50-4ade-9070-d0f223932e12-kube-api-access-zjjtt\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.803953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.803955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.804154 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.803982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.804154 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.804032 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.804154 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.804057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.804154 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.804085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d83067e-4a50-4ade-9070-d0f223932e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d83067e-4a50-4ade-9070-d0f223932e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjtt\" (UniqueName: \"kubernetes.io/projected/1d83067e-4a50-4ade-9070-d0f223932e12-kube-api-access-zjjtt\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.904747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.904693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.905674 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.905646 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d83067e-4a50-4ade-9070-d0f223932e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.907369 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.907344 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-ca\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.907460 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.907352 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.907524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.907508 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.907599 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.907578 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d83067e-4a50-4ade-9070-d0f223932e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:23.912941 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:23.912907 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjtt\" (UniqueName: \"kubernetes.io/projected/1d83067e-4a50-4ade-9070-d0f223932e12-kube-api-access-zjjtt\") pod \"cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8\" (UID: \"1d83067e-4a50-4ade-9070-d0f223932e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:24.036521 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:24.036434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" Jul 09 12:16:24.174905 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:24.174821 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8"] Jul 09 12:16:24.177390 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:16:24.177362 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d83067e_4a50_4ade_9070_d0f223932e12.slice/crio-22bedc27cbaca465eb66c6d08920a44fab3ec2a87d307671cec2fb29d4c3082b WatchSource:0}: Error finding container 22bedc27cbaca465eb66c6d08920a44fab3ec2a87d307671cec2fb29d4c3082b: Status 404 returned error can't find the container with id 22bedc27cbaca465eb66c6d08920a44fab3ec2a87d307671cec2fb29d4c3082b Jul 09 12:16:24.619590 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:24.619549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" event={"ID":"1d83067e-4a50-4ade-9070-d0f223932e12","Type":"ContainerStarted","Data":"22bedc27cbaca465eb66c6d08920a44fab3ec2a87d307671cec2fb29d4c3082b"} Jul 09 12:16:28.640828 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:28.640789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" event={"ID":"1d83067e-4a50-4ade-9070-d0f223932e12","Type":"ContainerStarted","Data":"013e8ad328e626420764a7071197834b681281eb627e26a811d47990ac611ca6"} Jul 09 12:16:28.641283 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:28.640838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" event={"ID":"1d83067e-4a50-4ade-9070-d0f223932e12","Type":"ContainerStarted","Data":"4510ab1e86f15f81c7010de1a1acb5c28602154375eb2fb984c6dc4d55de19b0"} Jul 09 12:16:28.641283 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:28.640855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" event={"ID":"1d83067e-4a50-4ade-9070-d0f223932e12","Type":"ContainerStarted","Data":"00ac30201a4ea945ac31654a791cc6c5a5579e54c53958e548326d8dae8c69ae"} Jul 09 12:16:28.663575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:16:28.663514 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5b6c4dc7-n4fr8" podStartSLOduration=1.94612026 podStartE2EDuration="5.66349587s" totalImagesPullingTime="3.71737561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:16:28.660650483 +0000 UTC m=+823.423376019" watchObservedRunningTime="2026-07-09 12:16:28.66349587 +0000 UTC m=+823.426221407" Jul 09 12:17:35.752540 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.752461 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch"] Jul 09 12:17:35.755163 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.755143 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.758573 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.758547 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 12:17:35.758573 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.758561 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 12:17:35.758725 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.758619 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 12:17:35.758725 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.758632 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fzkqb\"" Jul 09 12:17:35.758967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.758949 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 12:17:35.773086 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.773062 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch"] Jul 09 12:17:35.804689 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.804656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.804865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.804700 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-tmp\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.804865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.804728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xck\" (UniqueName: \"kubernetes.io/projected/14f904c8-e496-4c30-815c-2d79bd502f9f-kube-api-access-k8xck\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.804865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.804752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.804865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.804861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-manifests\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.905417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.905386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.905609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.905432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-tmp\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.905609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.905463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xck\" (UniqueName: \"kubernetes.io/projected/14f904c8-e496-4c30-815c-2d79bd502f9f-kube-api-access-k8xck\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.905609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.905487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.905789 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.905727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-manifests\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.906178 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.906154 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-manifests\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.908113 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.908092 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.908478 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.908448 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f904c8-e496-4c30-815c-2d79bd502f9f-tmp\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.908620 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.908602 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f904c8-e496-4c30-815c-2d79bd502f9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:35.915530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:35.915507 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xck\" (UniqueName: \"kubernetes.io/projected/14f904c8-e496-4c30-815c-2d79bd502f9f-kube-api-access-k8xck\") pod \"opendatahub-operator-controller-manager-7bd7989cf5-494ch\" (UID: \"14f904c8-e496-4c30-815c-2d79bd502f9f\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:36.082456 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:36.082374 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:36.220713 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:36.220688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch"] Jul 09 12:17:36.223368 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:17:36.223341 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f904c8_e496_4c30_815c_2d79bd502f9f.slice/crio-09b3bd3480ef2116974444e29e564cb3803ed4f13e1d1e0ce94d453f51283e42 WatchSource:0}: Error finding container 09b3bd3480ef2116974444e29e564cb3803ed4f13e1d1e0ce94d453f51283e42: Status 404 returned error can't find the container with id 09b3bd3480ef2116974444e29e564cb3803ed4f13e1d1e0ce94d453f51283e42 Jul 09 12:17:36.896347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:36.896310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" event={"ID":"14f904c8-e496-4c30-815c-2d79bd502f9f","Type":"ContainerStarted","Data":"09b3bd3480ef2116974444e29e564cb3803ed4f13e1d1e0ce94d453f51283e42"} Jul 09 12:17:39.912769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:39.912724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" event={"ID":"14f904c8-e496-4c30-815c-2d79bd502f9f","Type":"ContainerStarted","Data":"f7b3723e48951e2fa3f06eb95034ce163fb8a6d82e893e4e0f419d21dfcfc41d"} Jul 09 12:17:40.918539 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:40.918439 2567 generic.go:356] "Generic (PLEG): container finished" podID="14f904c8-e496-4c30-815c-2d79bd502f9f" containerID="f7b3723e48951e2fa3f06eb95034ce163fb8a6d82e893e4e0f419d21dfcfc41d" exitCode=0 Jul 09 12:17:40.918954 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:40.918537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" event={"ID":"14f904c8-e496-4c30-815c-2d79bd502f9f","Type":"ContainerDied","Data":"f7b3723e48951e2fa3f06eb95034ce163fb8a6d82e893e4e0f419d21dfcfc41d"} Jul 09 12:17:41.924955 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:41.924899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" event={"ID":"14f904c8-e496-4c30-815c-2d79bd502f9f","Type":"ContainerStarted","Data":"57ca711e3f094000ba07bf1ca25f903d47e2b9f8eabde03c9435ad0012c4ae02"} Jul 09 12:17:41.925387 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:41.925009 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:41.950040 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:41.949987 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" podStartSLOduration=2.333909399 podStartE2EDuration="6.949919101s" totalImagesPullingTime="3.52886103s" totalInitContainerRuntime="1.087148672s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:17:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:17:41.94939665 +0000 UTC m=+896.712122195" watchObservedRunningTime="2026-07-09 12:17:41.949919101 +0000 UTC m=+896.712644638" Jul 09 12:17:52.931417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:52.931388 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd7989cf5-494ch" Jul 09 12:17:55.912005 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.911972 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr"] Jul 09 12:17:55.922902 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.922870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:55.925458 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.925434 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 12:17:55.925596 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.925462 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 12:17:55.925596 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.925444 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 12:17:55.926252 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.926236 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 12:17:55.935377 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:55.935356 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr"] Jul 09 12:17:56.039751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.039717 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq8hd"] Jul 09 12:17:56.042118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.042097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.045380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.045357 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 12:17:56.045614 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.045450 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7ml5q\"" Jul 09 12:17:56.059091 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.059065 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq8hd"] Jul 09 12:17:56.088492 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.088460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01e03dea-1908-4135-86a5-43d6d29c1a68-tmp\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.088668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.088531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cqk\" (UniqueName: \"kubernetes.io/projected/01e03dea-1908-4135-86a5-43d6d29c1a68-kube-api-access-q6cqk\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.088668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.088643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.189681 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.189647 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.189899 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:17:56.189767 2567 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 09 12:17:56.190000 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.189883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.190000 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:17:56.189983 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs podName:01e03dea-1908-4135-86a5-43d6d29c1a68 nodeName:}" failed. No retries permitted until 2026-07-09 12:17:56.689955703 +0000 UTC m=+911.452681230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs") pod "kube-auth-proxy-5cb4f5859d-lm6fr" (UID: "01e03dea-1908-4135-86a5-43d6d29c1a68") : secret "kube-auth-proxy-tls" not found Jul 09 12:17:56.190107 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.190024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mxql\" (UniqueName: \"kubernetes.io/projected/adb5a6bc-49b5-4c98-a343-a66acfce46b1-kube-api-access-6mxql\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.190107 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.190073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01e03dea-1908-4135-86a5-43d6d29c1a68-tmp\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.190207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.190116 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cqk\" (UniqueName: \"kubernetes.io/projected/01e03dea-1908-4135-86a5-43d6d29c1a68-kube-api-access-q6cqk\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.192551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.192524 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01e03dea-1908-4135-86a5-43d6d29c1a68-tmp\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.202555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.202531 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cqk\" (UniqueName: \"kubernetes.io/projected/01e03dea-1908-4135-86a5-43d6d29c1a68-kube-api-access-q6cqk\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.291094 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.291056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.291094 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.291098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mxql\" (UniqueName: \"kubernetes.io/projected/adb5a6bc-49b5-4c98-a343-a66acfce46b1-kube-api-access-6mxql\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.291342 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:17:56.291226 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 12:17:56.291342 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:17:56.291300 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert podName:adb5a6bc-49b5-4c98-a343-a66acfce46b1 nodeName:}" failed. No retries permitted until 2026-07-09 12:17:56.791278954 +0000 UTC m=+911.554004484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert") pod "odh-model-controller-7998c67dc4-mq8hd" (UID: "adb5a6bc-49b5-4c98-a343-a66acfce46b1") : secret "odh-model-controller-webhook-cert" not found Jul 09 12:17:56.300718 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.300692 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mxql\" (UniqueName: \"kubernetes.io/projected/adb5a6bc-49b5-4c98-a343-a66acfce46b1-kube-api-access-6mxql\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.694944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.694889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.697594 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.697568 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e03dea-1908-4135-86a5-43d6d29c1a68-tls-certs\") pod \"kube-auth-proxy-5cb4f5859d-lm6fr\" (UID: \"01e03dea-1908-4135-86a5-43d6d29c1a68\") " pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.753082 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.753052 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mkwdm"] Jul 09 12:17:56.755784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.755764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:56.758762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.758743 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 09 12:17:56.780145 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.780121 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mkwdm"] Jul 09 12:17:56.796157 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.796124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.798766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.798736 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adb5a6bc-49b5-4c98-a343-a66acfce46b1-cert\") pod \"odh-model-controller-7998c67dc4-mq8hd\" (UID: \"adb5a6bc-49b5-4c98-a343-a66acfce46b1\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.833942 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.833881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" Jul 09 12:17:56.897099 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.896903 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-tls-certs\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:56.897099 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.897060 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dc2d\" (UniqueName: \"kubernetes.io/projected/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-kube-api-access-4dc2d\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:56.956333 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.956225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:17:56.976951 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.976907 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr"] Jul 09 12:17:56.979041 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:17:56.979006 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e03dea_1908_4135_86a5_43d6d29c1a68.slice/crio-a890002eecaaaaddcef91f3ada45b9e6c02552afde27855217082797595fa488 WatchSource:0}: Error finding container a890002eecaaaaddcef91f3ada45b9e6c02552afde27855217082797595fa488: Status 404 returned error can't find the container with id a890002eecaaaaddcef91f3ada45b9e6c02552afde27855217082797595fa488 Jul 09 12:17:56.993712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.993671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" event={"ID":"01e03dea-1908-4135-86a5-43d6d29c1a68","Type":"ContainerStarted","Data":"a890002eecaaaaddcef91f3ada45b9e6c02552afde27855217082797595fa488"} Jul 09 12:17:56.998303 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.998271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-tls-certs\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:56.998446 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:56.998318 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dc2d\" (UniqueName: \"kubernetes.io/projected/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-kube-api-access-4dc2d\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:57.002441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.002414 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-tls-certs\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:57.012657 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.012327 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dc2d\" (UniqueName: \"kubernetes.io/projected/afe8e85c-dd8f-4d1c-bbb9-3b665410c59f-kube-api-access-4dc2d\") pod \"model-serving-api-54d89bd5fc-mkwdm\" (UID: \"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f\") " pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:57.066179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.066145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:17:57.116336 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.116306 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq8hd"] Jul 09 12:17:57.118139 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:17:57.118112 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb5a6bc_49b5_4c98_a343_a66acfce46b1.slice/crio-0c16a5dadd6b6867300bad66e4fa1ad8e5d7b60fbf44b0489cdb99f568fb7089 WatchSource:0}: Error finding container 0c16a5dadd6b6867300bad66e4fa1ad8e5d7b60fbf44b0489cdb99f568fb7089: Status 404 returned error can't find the container with id 0c16a5dadd6b6867300bad66e4fa1ad8e5d7b60fbf44b0489cdb99f568fb7089 Jul 09 12:17:57.231062 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.230975 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mkwdm"] Jul 09 12:17:57.233314 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:17:57.233286 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe8e85c_dd8f_4d1c_bbb9_3b665410c59f.slice/crio-e66e6370c70447d979adb2aeab8616c54a0ad1960230da7d383d673dc7aabd8b WatchSource:0}: Error finding container e66e6370c70447d979adb2aeab8616c54a0ad1960230da7d383d673dc7aabd8b: Status 404 returned error can't find the container with id e66e6370c70447d979adb2aeab8616c54a0ad1960230da7d383d673dc7aabd8b Jul 09 12:17:57.640501 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.640424 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:17:57.643614 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.643599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.646533 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.646514 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 12:17:57.646657 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.646533 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 12:17:57.646657 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.646533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 12:17:57.646803 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.646786 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qrh9p\"" Jul 09 12:17:57.646871 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.646858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 12:17:57.662894 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.662871 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:17:57.806703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.806890 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.806890 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806735 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s984\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.806890 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.806890 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806846 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.806890 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.807141 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.806896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908287 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908287 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s984\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.908511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.908446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.909041 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.909005 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.910912 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.910881 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.911199 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.911169 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.911289 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.911260 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.911471 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.911449 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.921697 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.921672 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s984\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.921959 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.921916 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token\") pod \"istiod-openshift-gateway-7574459656-gk6cs\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:57.953786 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:57.953755 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:17:58.006683 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:58.006637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" event={"ID":"adb5a6bc-49b5-4c98-a343-a66acfce46b1","Type":"ContainerStarted","Data":"0c16a5dadd6b6867300bad66e4fa1ad8e5d7b60fbf44b0489cdb99f568fb7089"} Jul 09 12:17:58.009574 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:58.009537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" event={"ID":"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f","Type":"ContainerStarted","Data":"e66e6370c70447d979adb2aeab8616c54a0ad1960230da7d383d673dc7aabd8b"} Jul 09 12:17:58.142648 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:58.142620 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:17:58.145753 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:17:58.145715 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabecdbfc_08c9_4ba7_af92_1248980e2668.slice/crio-6aacdc640f232e3f21a733a7cc09d85501bae69175c0d23bab555eff66a779bb WatchSource:0}: Error finding container 6aacdc640f232e3f21a733a7cc09d85501bae69175c0d23bab555eff66a779bb: Status 404 returned error can't find the container with id 6aacdc640f232e3f21a733a7cc09d85501bae69175c0d23bab555eff66a779bb Jul 09 12:17:59.018686 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:17:59.018606 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" event={"ID":"abecdbfc-08c9-4ba7-af92-1248980e2668","Type":"ContainerStarted","Data":"6aacdc640f232e3f21a733a7cc09d85501bae69175c0d23bab555eff66a779bb"} Jul 09 12:18:02.038478 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.038437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" event={"ID":"adb5a6bc-49b5-4c98-a343-a66acfce46b1","Type":"ContainerStarted","Data":"1be5e1b67ae90c9f4e0dce960b2d38d0787144ffebcaf077ae205a2d488e332f"} Jul 09 12:18:02.038951 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.038532 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:18:02.040550 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.040520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" event={"ID":"afe8e85c-dd8f-4d1c-bbb9-3b665410c59f","Type":"ContainerStarted","Data":"57da482834f2149c5d1692b94ab8eaa2f773fe961aa8069e85e4f174f1edec85"} Jul 09 12:18:02.040681 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.040578 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:18:02.042310 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.042287 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" event={"ID":"01e03dea-1908-4135-86a5-43d6d29c1a68","Type":"ContainerStarted","Data":"d07163d75a9282a349e9a33fe7dfdea5d84c049314cc4e41ab2816330e807fe4"} Jul 09 12:18:02.055366 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.055314 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" podStartSLOduration=1.8345304279999999 podStartE2EDuration="6.055294538s" totalImagesPullingTime="4.22076411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:18:02.054962701 +0000 UTC m=+916.817688258" watchObservedRunningTime="2026-07-09 12:18:02.055294538 +0000 UTC m=+916.818020079" Jul 09 12:18:02.072704 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.072647 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb4f5859d-lm6fr" podStartSLOduration=2.71450892 podStartE2EDuration="7.072624202s" totalImagesPullingTime="4.358115282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:18:02.070401557 +0000 UTC m=+916.833127121" watchObservedRunningTime="2026-07-09 12:18:02.072624202 +0000 UTC m=+916.835349742" Jul 09 12:18:02.090596 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:02.090275 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" podStartSLOduration=1.9957045610000002 podStartE2EDuration="6.090257162s" totalImagesPullingTime="4.094552601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:18:02.08842319 +0000 UTC m=+916.851148752" watchObservedRunningTime="2026-07-09 12:18:02.090257162 +0000 UTC m=+916.852982699" Jul 09 12:18:03.264001 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:03.263960 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:18:03.264294 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:03.264031 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:18:04.055104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:04.055057 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" event={"ID":"abecdbfc-08c9-4ba7-af92-1248980e2668","Type":"ContainerStarted","Data":"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d"} Jul 09 12:18:04.055336 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:04.055231 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:18:04.057186 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:04.057156 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-gk6cs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 12:18:04.057186 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:04.057213 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 12:18:04.088042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:04.087979 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" podStartSLOduration=1.97179348 podStartE2EDuration="7.087955619s" totalImagesPullingTime="5.116162139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:18:04.084885294 +0000 UTC m=+918.847610868" watchObservedRunningTime="2026-07-09 12:18:04.087955619 +0000 UTC m=+918.850681156" Jul 09 12:18:05.060380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:05.060345 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:18:07.414602 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.414569 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65"] Jul 09 12:18:07.417183 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.417166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:07.421701 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.421680 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 12:18:07.421811 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.421681 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kbrc5\"" Jul 09 12:18:07.437922 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.437896 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65"] Jul 09 12:18:07.499227 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.499194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:07.499408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.499246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgq4\" (UniqueName: \"kubernetes.io/projected/1822ec44-47df-4500-8692-0c5950d2015f-kube-api-access-csgq4\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:07.600610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.600551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:07.600610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.600614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgq4\" (UniqueName: \"kubernetes.io/projected/1822ec44-47df-4500-8692-0c5950d2015f-kube-api-access-csgq4\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:07.600985 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:18:07.600760 2567 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 12:18:07.600985 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:18:07.600834 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert podName:1822ec44-47df-4500-8692-0c5950d2015f nodeName:}" failed. No retries permitted until 2026-07-09 12:18:08.100810121 +0000 UTC m=+922.863535649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert") pod "llmisvc-controller-manager-777c6bd7d5-tsl65" (UID: "1822ec44-47df-4500-8692-0c5950d2015f") : secret "llmisvc-webhook-server-cert" not found Jul 09 12:18:07.616755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:07.616710 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgq4\" (UniqueName: \"kubernetes.io/projected/1822ec44-47df-4500-8692-0c5950d2015f-kube-api-access-csgq4\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:08.106419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:08.106385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:08.108911 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:08.108876 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1822ec44-47df-4500-8692-0c5950d2015f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tsl65\" (UID: \"1822ec44-47df-4500-8692-0c5950d2015f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:08.327168 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:08.327127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:08.473974 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:08.470616 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65"] Jul 09 12:18:08.475638 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:18:08.475606 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1822ec44_47df_4500_8692_0c5950d2015f.slice/crio-6e39624e3f6f085b272d5e757b6d90704b6296eb609c520315e7f1d734a9deca WatchSource:0}: Error finding container 6e39624e3f6f085b272d5e757b6d90704b6296eb609c520315e7f1d734a9deca: Status 404 returned error can't find the container with id 6e39624e3f6f085b272d5e757b6d90704b6296eb609c520315e7f1d734a9deca Jul 09 12:18:09.078381 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:09.078337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" event={"ID":"1822ec44-47df-4500-8692-0c5950d2015f","Type":"ContainerStarted","Data":"6e39624e3f6f085b272d5e757b6d90704b6296eb609c520315e7f1d734a9deca"} Jul 09 12:18:09.724647 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:09.724607 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:18:09.725073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:09.724820 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerName="discovery" containerID="cri-o://c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d" gracePeriod=30 Jul 09 12:18:09.979978 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:09.979906 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:18:10.023096 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023061 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.023269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023115 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.023269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.023269 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023202 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s984\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.024553 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023311 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.024553 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023345 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.024553 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.023369 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap\") pod \"abecdbfc-08c9-4ba7-af92-1248980e2668\" (UID: \"abecdbfc-08c9-4ba7-af92-1248980e2668\") " Jul 09 12:18:10.024553 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.024141 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:18:10.026560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.026531 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts" (OuterVolumeSpecName: "cacerts") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:18:10.026978 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.026897 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs" (OuterVolumeSpecName: "local-certs") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:18:10.027882 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.027847 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:18:10.028376 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.028346 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:18:10.029327 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.029304 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984" (OuterVolumeSpecName: "kube-api-access-6s984") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "kube-api-access-6s984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:18:10.029409 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.029390 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token" (OuterVolumeSpecName: "istio-token") pod "abecdbfc-08c9-4ba7-af92-1248980e2668" (UID: "abecdbfc-08c9-4ba7-af92-1248980e2668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:18:10.085666 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.085618 2567 generic.go:356] "Generic (PLEG): container finished" podID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerID="c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d" exitCode=0 Jul 09 12:18:10.085862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.085754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" event={"ID":"abecdbfc-08c9-4ba7-af92-1248980e2668","Type":"ContainerDied","Data":"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d"} Jul 09 12:18:10.085862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.085801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" event={"ID":"abecdbfc-08c9-4ba7-af92-1248980e2668","Type":"ContainerDied","Data":"6aacdc640f232e3f21a733a7cc09d85501bae69175c0d23bab555eff66a779bb"} Jul 09 12:18:10.085862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.085828 2567 scope.go:117] "RemoveContainer" containerID="c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d" Jul 09 12:18:10.086080 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.085947 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs" Jul 09 12:18:10.114345 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.114303 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:18:10.124278 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124243 2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-cacerts\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124278 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124281 2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-kubeconfig\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124298 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s984\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-kube-api-access-6s984\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124313 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-dns-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124327 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124340 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abecdbfc-08c9-4ba7-af92-1248980e2668-istio-csr-ca-configmap\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.124515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.124353 2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abecdbfc-08c9-4ba7-af92-1248980e2668-local-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:18:10.125869 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.125831 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-gk6cs"] Jul 09 12:18:10.158440 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.158414 2567 scope.go:117] "RemoveContainer" containerID="c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d" Jul 09 12:18:10.158972 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:18:10.158945 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d\": container with ID starting with c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d not found: ID does not exist" containerID="c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d" Jul 09 12:18:10.159054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:10.158986 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d"} err="failed to get container status \"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d\": rpc error: code = NotFound desc = could not find container \"c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d\": container with ID starting with c9601fb382818c3b840fd6a4cc87186d227cd3f57b50205d4ad14e02ec73f71d not found: ID does not exist" Jul 09 12:18:11.848989 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:11.848956 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" path="/var/lib/kubelet/pods/abecdbfc-08c9-4ba7-af92-1248980e2668/volumes" Jul 09 12:18:13.050180 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:13.050152 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mq8hd" Jul 09 12:18:13.052172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:13.052155 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mkwdm" Jul 09 12:18:13.106500 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:13.106444 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" event={"ID":"1822ec44-47df-4500-8692-0c5950d2015f","Type":"ContainerStarted","Data":"c0961384011d71e2b37806fe123838e4dda83a4069f98531c74ca6ec59a3f413"} Jul 09 12:18:13.106776 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:13.106760 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:13.122771 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:13.122722 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" podStartSLOduration=2.414601993 podStartE2EDuration="6.122707453s" totalImagesPullingTime="3.70810546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:18:13.122024554 +0000 UTC m=+927.884750118" watchObservedRunningTime="2026-07-09 12:18:13.122707453 +0000 UTC m=+927.885432989" Jul 09 12:18:44.113421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:44.113388 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tsl65" Jul 09 12:18:56.831633 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.831599 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:18:56.832051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.831954 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerName="discovery" Jul 09 12:18:56.832051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.831966 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerName="discovery" Jul 09 12:18:56.832051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.832038 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="abecdbfc-08c9-4ba7-af92-1248980e2668" containerName="discovery" Jul 09 12:18:56.835186 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.835166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:18:56.837916 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.837892 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 12:18:56.838618 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.838600 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 12:18:56.838618 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.838609 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-75f5j\"" Jul 09 12:18:56.844788 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.843917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:18:56.946091 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:56.946058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwjr\" (UniqueName: \"kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr\") pod \"kuadrant-operator-catalog-5cgxc\" (UID: \"37a3e4b8-f74d-4d8d-b3c9-309004ae161e\") " pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:18:57.047175 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.047145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwjr\" (UniqueName: \"kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr\") pod \"kuadrant-operator-catalog-5cgxc\" (UID: \"37a3e4b8-f74d-4d8d-b3c9-309004ae161e\") " pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:18:57.056016 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.055988 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwjr\" (UniqueName: \"kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr\") pod \"kuadrant-operator-catalog-5cgxc\" (UID: \"37a3e4b8-f74d-4d8d-b3c9-309004ae161e\") " pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:18:57.148139 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.148062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:18:57.173808 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.173782 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:18:57.278882 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.278852 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:18:57.281395 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:18:57.281364 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a3e4b8_f74d_4d8d_b3c9_309004ae161e.slice/crio-c6dffb08c2a085ad88d453dac3028cd465f723e1ab1cb637eda95d302f38c205 WatchSource:0}: Error finding container c6dffb08c2a085ad88d453dac3028cd465f723e1ab1cb637eda95d302f38c205: Status 404 returned error can't find the container with id c6dffb08c2a085ad88d453dac3028cd465f723e1ab1cb637eda95d302f38c205 Jul 09 12:18:57.282723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.282697 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:18:57.304828 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.304805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" event={"ID":"37a3e4b8-f74d-4d8d-b3c9-309004ae161e","Type":"ContainerStarted","Data":"c6dffb08c2a085ad88d453dac3028cd465f723e1ab1cb637eda95d302f38c205"} Jul 09 12:18:57.385186 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.385160 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pd6qz"] Jul 09 12:18:57.388255 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.388240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:18:57.396673 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.396653 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pd6qz"] Jul 09 12:18:57.450597 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.450569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj6l\" (UniqueName: \"kubernetes.io/projected/9e37f6d4-c593-40cc-9e14-2c3b4382ec81-kube-api-access-9gj6l\") pod \"kuadrant-operator-catalog-pd6qz\" (UID: \"9e37f6d4-c593-40cc-9e14-2c3b4382ec81\") " pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:18:57.551686 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.551646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj6l\" (UniqueName: \"kubernetes.io/projected/9e37f6d4-c593-40cc-9e14-2c3b4382ec81-kube-api-access-9gj6l\") pod \"kuadrant-operator-catalog-pd6qz\" (UID: \"9e37f6d4-c593-40cc-9e14-2c3b4382ec81\") " pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:18:57.559531 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.559509 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj6l\" (UniqueName: \"kubernetes.io/projected/9e37f6d4-c593-40cc-9e14-2c3b4382ec81-kube-api-access-9gj6l\") pod \"kuadrant-operator-catalog-pd6qz\" (UID: \"9e37f6d4-c593-40cc-9e14-2c3b4382ec81\") " pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:18:57.698482 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.698436 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:18:57.830894 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:57.830718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pd6qz"] Jul 09 12:18:57.833105 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:18:57.833079 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e37f6d4_c593_40cc_9e14_2c3b4382ec81.slice/crio-67fec6a0a2fb8482a71d33012b6fe7334aa10a36dc17777a022507494a8d1ac1 WatchSource:0}: Error finding container 67fec6a0a2fb8482a71d33012b6fe7334aa10a36dc17777a022507494a8d1ac1: Status 404 returned error can't find the container with id 67fec6a0a2fb8482a71d33012b6fe7334aa10a36dc17777a022507494a8d1ac1 Jul 09 12:18:58.310547 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:18:58.310511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" event={"ID":"9e37f6d4-c593-40cc-9e14-2c3b4382ec81","Type":"ContainerStarted","Data":"67fec6a0a2fb8482a71d33012b6fe7334aa10a36dc17777a022507494a8d1ac1"} Jul 09 12:19:00.323044 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.322922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" event={"ID":"9e37f6d4-c593-40cc-9e14-2c3b4382ec81","Type":"ContainerStarted","Data":"835352fc7cb44bf47d4f1a56a185e14e209ba51455e7ab820b95f687e8f2acb9"} Jul 09 12:19:00.324476 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.324453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" event={"ID":"37a3e4b8-f74d-4d8d-b3c9-309004ae161e","Type":"ContainerStarted","Data":"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6"} Jul 09 12:19:00.324623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.324521 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" podUID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" containerName="registry-server" containerID="cri-o://042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6" gracePeriod=2 Jul 09 12:19:00.339337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.339298 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" podStartSLOduration=1.16386608 podStartE2EDuration="3.339280757s" totalImagesPullingTime="2.175414677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:00.339178258 +0000 UTC m=+975.101903815" watchObservedRunningTime="2026-07-09 12:19:00.339280757 +0000 UTC m=+975.102006294" Jul 09 12:19:00.356620 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.356579 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" podStartSLOduration=1.6255692659999998 podStartE2EDuration="4.356565846s" totalImagesPullingTime="2.73099658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:00.35600579 +0000 UTC m=+975.118731366" watchObservedRunningTime="2026-07-09 12:19:00.356565846 +0000 UTC m=+975.119291389" Jul 09 12:19:00.571382 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.571359 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:19:00.678724 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.678659 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpwjr\" (UniqueName: \"kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr\") pod \"37a3e4b8-f74d-4d8d-b3c9-309004ae161e\" (UID: \"37a3e4b8-f74d-4d8d-b3c9-309004ae161e\") " Jul 09 12:19:00.680818 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.680790 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr" (OuterVolumeSpecName: "kube-api-access-lpwjr") pod "37a3e4b8-f74d-4d8d-b3c9-309004ae161e" (UID: "37a3e4b8-f74d-4d8d-b3c9-309004ae161e"). InnerVolumeSpecName "kube-api-access-lpwjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:19:00.779439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:00.779412 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwjr\" (UniqueName: \"kubernetes.io/projected/37a3e4b8-f74d-4d8d-b3c9-309004ae161e-kube-api-access-lpwjr\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:19:01.330140 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.330050 2567 generic.go:356] "Generic (PLEG): container finished" podID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" containerID="042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6" exitCode=0 Jul 09 12:19:01.330140 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.330103 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" Jul 09 12:19:01.330555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.330147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" event={"ID":"37a3e4b8-f74d-4d8d-b3c9-309004ae161e","Type":"ContainerDied","Data":"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6"} Jul 09 12:19:01.330555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.330194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5cgxc" event={"ID":"37a3e4b8-f74d-4d8d-b3c9-309004ae161e","Type":"ContainerDied","Data":"c6dffb08c2a085ad88d453dac3028cd465f723e1ab1cb637eda95d302f38c205"} Jul 09 12:19:01.330555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.330220 2567 scope.go:117] "RemoveContainer" containerID="042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6" Jul 09 12:19:01.345388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.345365 2567 scope.go:117] "RemoveContainer" containerID="042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6" Jul 09 12:19:01.345696 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:01.345674 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6\": container with ID starting with 042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6 not found: ID does not exist" containerID="042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6" Jul 09 12:19:01.345746 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.345704 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6"} err="failed to get container status \"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6\": rpc error: code = NotFound desc = could not find container \"042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6\": container with ID starting with 042f2a1f8f186e8dddf72728976349ce87a4cbaf4f4f20b92218c10b6edc88a6 not found: ID does not exist" Jul 09 12:19:01.352631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.352609 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:19:01.358678 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.358657 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5cgxc"] Jul 09 12:19:01.847371 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:01.847339 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" path="/var/lib/kubelet/pods/37a3e4b8-f74d-4d8d-b3c9-309004ae161e/volumes" Jul 09 12:19:07.698790 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:07.698705 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:19:07.699218 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:07.698798 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:19:07.721859 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:07.721836 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:19:08.383002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:08.382971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-pd6qz" Jul 09 12:19:30.366719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.366683 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m"] Jul 09 12:19:30.367200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.367098 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" containerName="registry-server" Jul 09 12:19:30.367200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.367111 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" containerName="registry-server" Jul 09 12:19:30.367200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.367170 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a3e4b8-f74d-4d8d-b3c9-309004ae161e" containerName="registry-server" Jul 09 12:19:30.374475 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.374459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.377111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.377089 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qstkz\"" Jul 09 12:19:30.382724 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.382702 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m"] Jul 09 12:19:30.432207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.432180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg58\" (UniqueName: \"kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.432337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.432224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.532643 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.532611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.532794 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.532699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg58\" (UniqueName: \"kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.533000 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.532983 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.542640 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.542613 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg58\" (UniqueName: \"kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.685385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.685363 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:30.817274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:30.817191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m"] Jul 09 12:19:30.819580 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:30.819550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e977fe_d78b_4291_916c_a6b658e9b359.slice/crio-16fd52a125367a834c5b0423c763e3ded139adc2168b8f4a7322fdbbdac34d5c WatchSource:0}: Error finding container 16fd52a125367a834c5b0423c763e3ded139adc2168b8f4a7322fdbbdac34d5c: Status 404 returned error can't find the container with id 16fd52a125367a834c5b0423c763e3ded139adc2168b8f4a7322fdbbdac34d5c Jul 09 12:19:31.469860 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:31.469825 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" event={"ID":"23e977fe-d78b-4291-916c-a6b658e9b359","Type":"ContainerStarted","Data":"16fd52a125367a834c5b0423c763e3ded139adc2168b8f4a7322fdbbdac34d5c"} Jul 09 12:19:32.379001 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.378968 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf"] Jul 09 12:19:32.382769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.382746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:32.385719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.385697 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wjnwx\"" Jul 09 12:19:32.385719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.385710 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 12:19:32.394205 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.394185 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf"] Jul 09 12:19:32.450077 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.450046 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp24\" (UniqueName: \"kubernetes.io/projected/69e00e53-6251-4776-a221-0e166d68a434-kube-api-access-xfp24\") pod \"dns-operator-controller-manager-864dd78c4c-j2vdf\" (UID: \"69e00e53-6251-4776-a221-0e166d68a434\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:32.550790 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.550761 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp24\" (UniqueName: \"kubernetes.io/projected/69e00e53-6251-4776-a221-0e166d68a434-kube-api-access-xfp24\") pod \"dns-operator-controller-manager-864dd78c4c-j2vdf\" (UID: \"69e00e53-6251-4776-a221-0e166d68a434\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:32.559778 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.559755 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp24\" (UniqueName: \"kubernetes.io/projected/69e00e53-6251-4776-a221-0e166d68a434-kube-api-access-xfp24\") pod \"dns-operator-controller-manager-864dd78c4c-j2vdf\" (UID: \"69e00e53-6251-4776-a221-0e166d68a434\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:32.695088 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.695057 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:32.832850 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:32.832826 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf"] Jul 09 12:19:32.834414 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:32.834382 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e00e53_6251_4776_a221_0e166d68a434.slice/crio-5d2d843c5272716bc6bcbf5fda96a103fc5c4acc9079a1372f0255b12f068816 WatchSource:0}: Error finding container 5d2d843c5272716bc6bcbf5fda96a103fc5c4acc9079a1372f0255b12f068816: Status 404 returned error can't find the container with id 5d2d843c5272716bc6bcbf5fda96a103fc5c4acc9079a1372f0255b12f068816 Jul 09 12:19:33.481259 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:33.481222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" event={"ID":"69e00e53-6251-4776-a221-0e166d68a434","Type":"ContainerStarted","Data":"5d2d843c5272716bc6bcbf5fda96a103fc5c4acc9079a1372f0255b12f068816"} Jul 09 12:19:35.492571 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.492537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" event={"ID":"69e00e53-6251-4776-a221-0e166d68a434","Type":"ContainerStarted","Data":"3352bae9804ada687dd06ed49817ec853c9fd46823bcea255caf376f8674d455"} Jul 09 12:19:35.493143 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.492681 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:35.517338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.517292 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" podStartSLOduration=1.226058756 podStartE2EDuration="3.517272707s" totalImagesPullingTime="2.291213951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:35.516369152 +0000 UTC m=+1010.279094710" watchObservedRunningTime="2026-07-09 12:19:35.517272707 +0000 UTC m=+1010.279998251" Jul 09 12:19:35.595593 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.595562 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8"] Jul 09 12:19:35.598881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.598861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:35.601342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.601323 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fhlrz\"" Jul 09 12:19:35.607741 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.607719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8"] Jul 09 12:19:35.679139 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.679072 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvrx\" (UniqueName: \"kubernetes.io/projected/afa3a7e9-edb6-4c55-97eb-45fcc3594b48-kube-api-access-dpvrx\") pod \"authorino-operator-75cc8d5cb7-bzlz8\" (UID: \"afa3a7e9-edb6-4c55-97eb-45fcc3594b48\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:35.780605 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.780523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvrx\" (UniqueName: \"kubernetes.io/projected/afa3a7e9-edb6-4c55-97eb-45fcc3594b48-kube-api-access-dpvrx\") pod \"authorino-operator-75cc8d5cb7-bzlz8\" (UID: \"afa3a7e9-edb6-4c55-97eb-45fcc3594b48\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:35.789474 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.789443 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvrx\" (UniqueName: \"kubernetes.io/projected/afa3a7e9-edb6-4c55-97eb-45fcc3594b48-kube-api-access-dpvrx\") pod \"authorino-operator-75cc8d5cb7-bzlz8\" (UID: \"afa3a7e9-edb6-4c55-97eb-45fcc3594b48\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:35.909616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:35.909587 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:36.059081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:36.059012 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8"] Jul 09 12:19:36.062673 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:36.062648 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa3a7e9_edb6_4c55_97eb_45fcc3594b48.slice/crio-fc578bc5a8a6a3a062a631b03cb306ff8d11099a6484d3f7977a160dc3a90b0e WatchSource:0}: Error finding container fc578bc5a8a6a3a062a631b03cb306ff8d11099a6484d3f7977a160dc3a90b0e: Status 404 returned error can't find the container with id fc578bc5a8a6a3a062a631b03cb306ff8d11099a6484d3f7977a160dc3a90b0e Jul 09 12:19:36.499727 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:36.499683 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" event={"ID":"afa3a7e9-edb6-4c55-97eb-45fcc3594b48","Type":"ContainerStarted","Data":"fc578bc5a8a6a3a062a631b03cb306ff8d11099a6484d3f7977a160dc3a90b0e"} Jul 09 12:19:37.506535 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:37.506504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" event={"ID":"23e977fe-d78b-4291-916c-a6b658e9b359","Type":"ContainerStarted","Data":"71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a"} Jul 09 12:19:37.506901 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:37.506638 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:37.525674 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:37.525626 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" podStartSLOduration=0.905440249 podStartE2EDuration="7.525612771s" totalImagesPullingTime="6.620172522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:37.524232201 +0000 UTC m=+1012.286957759" watchObservedRunningTime="2026-07-09 12:19:37.525612771 +0000 UTC m=+1012.288338307" Jul 09 12:19:38.279483 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.279445 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69"] Jul 09 12:19:38.286072 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.286050 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:38.288551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.288530 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wskpq\"" Jul 09 12:19:38.294583 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.294565 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69"] Jul 09 12:19:38.303522 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.303492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfxx\" (UniqueName: \"kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx\") pod \"limitador-operator-controller-manager-859d7b78bd-76z69\" (UID: \"8725dc81-225c-47d6-820a-62dc9b6ef2ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:38.404950 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.404894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfxx\" (UniqueName: \"kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx\") pod \"limitador-operator-controller-manager-859d7b78bd-76z69\" (UID: \"8725dc81-225c-47d6-820a-62dc9b6ef2ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:38.415732 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.415710 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfxx\" (UniqueName: \"kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx\") pod \"limitador-operator-controller-manager-859d7b78bd-76z69\" (UID: \"8725dc81-225c-47d6-820a-62dc9b6ef2ba\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:38.598823 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:38.598745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:39.264762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:39.264733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69"] Jul 09 12:19:39.266235 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:39.266206 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8725dc81_225c_47d6_820a_62dc9b6ef2ba.slice/crio-ff82233cd2d6a01413b4fe1efe234132ce2d5a4a03617de41f96a864a3ce2362 WatchSource:0}: Error finding container ff82233cd2d6a01413b4fe1efe234132ce2d5a4a03617de41f96a864a3ce2362: Status 404 returned error can't find the container with id ff82233cd2d6a01413b4fe1efe234132ce2d5a4a03617de41f96a864a3ce2362 Jul 09 12:19:39.524414 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:39.524372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" event={"ID":"afa3a7e9-edb6-4c55-97eb-45fcc3594b48","Type":"ContainerStarted","Data":"01330596bdccf91921b1bf618510213c17c2ebcd9da08e3967cd7a8804e53d30"} Jul 09 12:19:39.524615 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:39.524488 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:39.525765 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:39.525743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" event={"ID":"8725dc81-225c-47d6-820a-62dc9b6ef2ba","Type":"ContainerStarted","Data":"ff82233cd2d6a01413b4fe1efe234132ce2d5a4a03617de41f96a864a3ce2362"} Jul 09 12:19:39.546672 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:39.546613 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" podStartSLOduration=1.423033366 podStartE2EDuration="4.54657408s" totalImagesPullingTime="3.123540714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:39.546451655 +0000 UTC m=+1014.309177192" watchObservedRunningTime="2026-07-09 12:19:39.54657408 +0000 UTC m=+1014.309299639" Jul 09 12:19:41.537247 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:41.537156 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" event={"ID":"8725dc81-225c-47d6-820a-62dc9b6ef2ba","Type":"ContainerStarted","Data":"bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb"} Jul 09 12:19:41.537596 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:41.537251 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:41.554169 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:41.554127 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" podStartSLOduration=1.5379924919999999 podStartE2EDuration="3.554113525s" totalImagesPullingTime="2.016121033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:41.553151885 +0000 UTC m=+1016.315877437" watchObservedRunningTime="2026-07-09 12:19:41.554113525 +0000 UTC m=+1016.316839118" Jul 09 12:19:46.502794 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:46.502757 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j2vdf" Jul 09 12:19:48.520304 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:48.520276 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.132566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.132531 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m"] Jul 09 12:19:50.133082 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.132783 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" containerName="manager" containerID="cri-o://71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a" gracePeriod=2 Jul 09 12:19:50.170560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.170523 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m"] Jul 09 12:19:50.172184 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.172159 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69"] Jul 09 12:19:50.172457 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.172411 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" containerName="manager" containerID="cri-o://bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb" gracePeriod=2 Jul 09 12:19:50.173949 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.173910 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:19:50.174659 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.174632 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" containerName="manager" Jul 09 12:19:50.174659 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.174661 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" containerName="manager" Jul 09 12:19:50.174821 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.174786 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" containerName="manager" Jul 09 12:19:50.178827 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.178807 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.179024 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.179012 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69"] Jul 09 12:19:50.179215 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.179194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.180303 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.180267 2567 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"23e977fe-d78b-4291-916c-a6b658e9b359\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\": pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.180661 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.180608 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:19:50.183294 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.183271 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl"] Jul 09 12:19:50.183816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.183796 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" containerName="manager" Jul 09 12:19:50.183816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.183816 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" containerName="manager" Jul 09 12:19:50.183988 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.183912 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" containerName="manager" Jul 09 12:19:50.187251 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.187229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:50.205059 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.205036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl"] Jul 09 12:19:50.212765 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.212733 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.223422 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.223390 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.225502 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.225477 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.227440 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.227419 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.304283 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.304249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.304422 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.304316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6jf\" (UniqueName: \"kubernetes.io/projected/6c25f385-ce0e-4d6c-8895-f819c3392468-kube-api-access-vj6jf\") pod \"limitador-operator-controller-manager-859d7b78bd-w7wsl\" (UID: \"6c25f385-ce0e-4d6c-8895-f819c3392468\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:50.304422 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.304356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxpr\" (UniqueName: \"kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.405275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.405247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6jf\" (UniqueName: \"kubernetes.io/projected/6c25f385-ce0e-4d6c-8895-f819c3392468-kube-api-access-vj6jf\") pod \"limitador-operator-controller-manager-859d7b78bd-w7wsl\" (UID: \"6c25f385-ce0e-4d6c-8895-f819c3392468\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:50.405367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.405296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxpr\" (UniqueName: \"kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.405367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.405357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.405671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.405653 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.414886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.414863 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6jf\" (UniqueName: \"kubernetes.io/projected/6c25f385-ce0e-4d6c-8895-f819c3392468-kube-api-access-vj6jf\") pod \"limitador-operator-controller-manager-859d7b78bd-w7wsl\" (UID: \"6c25f385-ce0e-4d6c-8895-f819c3392468\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:50.415699 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.415678 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxpr\" (UniqueName: \"kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cwmrf\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.416864 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.416841 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.420151 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.420134 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.420305 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.420280 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.422607 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.422583 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.424734 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.424714 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.426707 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.426686 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.506330 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.506307 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drfxx\" (UniqueName: \"kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx\") pod \"8725dc81-225c-47d6-820a-62dc9b6ef2ba\" (UID: \"8725dc81-225c-47d6-820a-62dc9b6ef2ba\") " Jul 09 12:19:50.506429 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.506338 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hg58\" (UniqueName: \"kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58\") pod \"23e977fe-d78b-4291-916c-a6b658e9b359\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " Jul 09 12:19:50.506429 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.506411 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume\") pod \"23e977fe-d78b-4291-916c-a6b658e9b359\" (UID: \"23e977fe-d78b-4291-916c-a6b658e9b359\") " Jul 09 12:19:50.506832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.506808 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23e977fe-d78b-4291-916c-a6b658e9b359" (UID: "23e977fe-d78b-4291-916c-a6b658e9b359"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:19:50.508511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.508490 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx" (OuterVolumeSpecName: "kube-api-access-drfxx") pod "8725dc81-225c-47d6-820a-62dc9b6ef2ba" (UID: "8725dc81-225c-47d6-820a-62dc9b6ef2ba"). InnerVolumeSpecName "kube-api-access-drfxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:19:50.508570 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.508490 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58" (OuterVolumeSpecName: "kube-api-access-7hg58") pod "23e977fe-d78b-4291-916c-a6b658e9b359" (UID: "23e977fe-d78b-4291-916c-a6b658e9b359"). InnerVolumeSpecName "kube-api-access-7hg58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:19:50.533349 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.533331 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bzlz8" Jul 09 12:19:50.535530 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.535510 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.554579 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.554553 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.575748 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.575728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:50.581512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.581492 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:50.584789 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.584764 2567 generic.go:356] "Generic (PLEG): container finished" podID="23e977fe-d78b-4291-916c-a6b658e9b359" containerID="71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a" exitCode=0 Jul 09 12:19:50.584865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.584811 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.584865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.584852 2567 scope.go:117] "RemoveContainer" containerID="71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a" Jul 09 12:19:50.586878 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.586787 2567 generic.go:356] "Generic (PLEG): container finished" podID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" containerID="bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb" exitCode=0 Jul 09 12:19:50.586878 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.586829 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.587322 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.587288 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.589042 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.589012 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.590814 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.590752 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.592423 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.592394 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.599984 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.599965 2567 scope.go:117] "RemoveContainer" containerID="71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a" Jul 09 12:19:50.600226 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.600196 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.600575 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.600443 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a\": container with ID starting with 71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a not found: ID does not exist" containerID="71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a" Jul 09 12:19:50.600575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.600477 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a"} err="failed to get container status \"71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a\": rpc error: code = NotFound desc = could not find container \"71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a\": container with ID starting with 71f3f09c0e80d3c3640b074b987d75879c356550da25633919ace3b1ae33186a not found: ID does not exist" Jul 09 12:19:50.600575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.600495 2567 scope.go:117] "RemoveContainer" containerID="bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb" Jul 09 12:19:50.602089 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.602035 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.604799 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.604767 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:50.606522 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.606478 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:50.608363 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.608117 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drfxx\" (UniqueName: \"kubernetes.io/projected/8725dc81-225c-47d6-820a-62dc9b6ef2ba-kube-api-access-drfxx\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:19:50.608363 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.608137 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hg58\" (UniqueName: \"kubernetes.io/projected/23e977fe-d78b-4291-916c-a6b658e9b359-kube-api-access-7hg58\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:19:50.608363 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.608147 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23e977fe-d78b-4291-916c-a6b658e9b359-extensions-socket-volume\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:19:50.610795 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.610777 2567 scope.go:117] "RemoveContainer" containerID="bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb" Jul 09 12:19:50.611126 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:50.611107 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb\": container with ID starting with bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb not found: ID does not exist" containerID="bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb" Jul 09 12:19:50.611207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.611138 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb"} err="failed to get container status \"bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb\": rpc error: code = NotFound desc = could not find container \"bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb\": container with ID starting with bbfd230d685d00c72436ab0ae2ff6b6689c932f6ce52cf7759190e1358133eeb not found: ID does not exist" Jul 09 12:19:50.725846 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.725812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:19:50.729302 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:50.729274 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cafbe7_aaa6_44b9_80db_9bb290c7a45d.slice/crio-63e9ac98ed8b4d35a38bf3b2d5b6cbebcccc896849f4c2be9dd243eef9ecb3d6 WatchSource:0}: Error finding container 63e9ac98ed8b4d35a38bf3b2d5b6cbebcccc896849f4c2be9dd243eef9ecb3d6: Status 404 returned error can't find the container with id 63e9ac98ed8b4d35a38bf3b2d5b6cbebcccc896849f4c2be9dd243eef9ecb3d6 Jul 09 12:19:50.745345 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:50.745321 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl"] Jul 09 12:19:50.748729 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:50.748707 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c25f385_ce0e_4d6c_8895_f819c3392468.slice/crio-5073d551e7211075c03dc7f956fea3a48518c670488074fbcceeac3348cd1088 WatchSource:0}: Error finding container 5073d551e7211075c03dc7f956fea3a48518c670488074fbcceeac3348cd1088: Status 404 returned error can't find the container with id 5073d551e7211075c03dc7f956fea3a48518c670488074fbcceeac3348cd1088 Jul 09 12:19:51.365921 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.365876 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:19:51.369749 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.369722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.374393 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.374368 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:51.377779 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.377754 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:51.384872 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.384848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:19:51.416200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.416143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjjh\" (UniqueName: \"kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.416200 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.416186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.517052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.517022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjjh\" (UniqueName: \"kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.517181 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.517078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.517482 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.517462 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.526291 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.526263 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjjh\" (UniqueName: \"kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh\") pod \"kuadrant-operator-controller-manager-86878c769f-mzkp2\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.593879 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.593850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" event={"ID":"85cafbe7-aaa6-44b9-80db-9bb290c7a45d","Type":"ContainerStarted","Data":"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f"} Jul 09 12:19:51.594063 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.593917 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:19:51.594063 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.593953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" event={"ID":"85cafbe7-aaa6-44b9-80db-9bb290c7a45d","Type":"ContainerStarted","Data":"63e9ac98ed8b4d35a38bf3b2d5b6cbebcccc896849f4c2be9dd243eef9ecb3d6"} Jul 09 12:19:51.595451 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.595427 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" event={"ID":"6c25f385-ce0e-4d6c-8895-f819c3392468","Type":"ContainerStarted","Data":"391db73e4429b1ec7e3a2e5f176487cc85ef75d7ca4aa143465be6bb76623ffc"} Jul 09 12:19:51.595556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.595474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" event={"ID":"6c25f385-ce0e-4d6c-8895-f819c3392468","Type":"ContainerStarted","Data":"5073d551e7211075c03dc7f956fea3a48518c670488074fbcceeac3348cd1088"} Jul 09 12:19:51.595556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.595497 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:19:51.614691 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.614656 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:51.615045 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.615010 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" podStartSLOduration=1.614998173 podStartE2EDuration="1.614998173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:51.612706554 +0000 UTC m=+1026.375432090" watchObservedRunningTime="2026-07-09 12:19:51.614998173 +0000 UTC m=+1026.377723709" Jul 09 12:19:51.616662 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.616639 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:51.640070 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.640032 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" podStartSLOduration=1.640020212 podStartE2EDuration="1.640020212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:51.638763203 +0000 UTC m=+1026.401488753" watchObservedRunningTime="2026-07-09 12:19:51.640020212 +0000 UTC m=+1026.402745748" Jul 09 12:19:51.640835 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.640814 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nrv6m\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrv6m" Jul 09 12:19:51.642738 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:19:51.642718 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76z69\" is forbidden: User \"system:node:ip-10-0-133-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-7.ec2.internal' and this object" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76z69" Jul 09 12:19:51.679999 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.679944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:51.825378 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.825352 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:19:51.827787 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:19:51.827746 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f35421_8d4e_4048_8c90_d0cfab2285bd.slice/crio-d70ae585a0419b75bec79f65b184b27d0c600517cab0e79e2724f8b531a453df WatchSource:0}: Error finding container d70ae585a0419b75bec79f65b184b27d0c600517cab0e79e2724f8b531a453df: Status 404 returned error can't find the container with id d70ae585a0419b75bec79f65b184b27d0c600517cab0e79e2724f8b531a453df Jul 09 12:19:51.861111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.861084 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e977fe-d78b-4291-916c-a6b658e9b359" path="/var/lib/kubelet/pods/23e977fe-d78b-4291-916c-a6b658e9b359/volumes" Jul 09 12:19:51.861519 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:51.861500 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8725dc81-225c-47d6-820a-62dc9b6ef2ba" path="/var/lib/kubelet/pods/8725dc81-225c-47d6-820a-62dc9b6ef2ba/volumes" Jul 09 12:19:52.604264 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:52.604222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" event={"ID":"b1f35421-8d4e-4048-8c90-d0cfab2285bd","Type":"ContainerStarted","Data":"6c089a5b540b4ef0e5417b1acd0ff2309c9a1c136ba996e4db19ea563aad9335"} Jul 09 12:19:52.604711 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:52.604281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" event={"ID":"b1f35421-8d4e-4048-8c90-d0cfab2285bd","Type":"ContainerStarted","Data":"d70ae585a0419b75bec79f65b184b27d0c600517cab0e79e2724f8b531a453df"} Jul 09 12:19:52.604711 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:52.604502 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:19:52.625702 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:19:52.625651 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" podStartSLOduration=1.625634389 podStartE2EDuration="1.625634389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:19:52.622583156 +0000 UTC m=+1027.385308693" watchObservedRunningTime="2026-07-09 12:19:52.625634389 +0000 UTC m=+1027.388359926" Jul 09 12:20:02.606111 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:02.606081 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w7wsl" Jul 09 12:20:02.606555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:02.606244 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:20:03.611786 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:03.611756 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:20:03.660834 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:03.660797 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:20:03.661051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:03.661024 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" podUID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" containerName="manager" containerID="cri-o://82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f" gracePeriod=10 Jul 09 12:20:03.914609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:03.914583 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:20:04.032079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.032047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume\") pod \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " Jul 09 12:20:04.032260 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.032114 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhxpr\" (UniqueName: \"kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr\") pod \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\" (UID: \"85cafbe7-aaa6-44b9-80db-9bb290c7a45d\") " Jul 09 12:20:04.032511 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.032487 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85cafbe7-aaa6-44b9-80db-9bb290c7a45d" (UID: "85cafbe7-aaa6-44b9-80db-9bb290c7a45d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:20:04.034326 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.034306 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr" (OuterVolumeSpecName: "kube-api-access-qhxpr") pod "85cafbe7-aaa6-44b9-80db-9bb290c7a45d" (UID: "85cafbe7-aaa6-44b9-80db-9bb290c7a45d"). InnerVolumeSpecName "kube-api-access-qhxpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:04.133524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.133484 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-extensions-socket-volume\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:04.133524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.133517 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhxpr\" (UniqueName: \"kubernetes.io/projected/85cafbe7-aaa6-44b9-80db-9bb290c7a45d-kube-api-access-qhxpr\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:04.663051 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.662990 2567 generic.go:356] "Generic (PLEG): container finished" podID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" containerID="82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f" exitCode=0 Jul 09 12:20:04.663418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.663070 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" Jul 09 12:20:04.663418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.663069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" event={"ID":"85cafbe7-aaa6-44b9-80db-9bb290c7a45d","Type":"ContainerDied","Data":"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f"} Jul 09 12:20:04.663418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.663146 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf" event={"ID":"85cafbe7-aaa6-44b9-80db-9bb290c7a45d","Type":"ContainerDied","Data":"63e9ac98ed8b4d35a38bf3b2d5b6cbebcccc896849f4c2be9dd243eef9ecb3d6"} Jul 09 12:20:04.663418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.663168 2567 scope.go:117] "RemoveContainer" containerID="82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f" Jul 09 12:20:04.672609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.672594 2567 scope.go:117] "RemoveContainer" containerID="82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f" Jul 09 12:20:04.672842 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:20:04.672823 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f\": container with ID starting with 82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f not found: ID does not exist" containerID="82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f" Jul 09 12:20:04.672896 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.672850 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f"} err="failed to get container status \"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f\": rpc error: code = NotFound desc = could not find container \"82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f\": container with ID starting with 82fa8be1fb6240fc7f6fadb9e40f5c5b6782ca12da2249707b477d54dd15133f not found: ID does not exist" Jul 09 12:20:04.690455 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.690434 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:20:04.697627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:04.697606 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cwmrf"] Jul 09 12:20:05.848906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:05.848873 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" path="/var/lib/kubelet/pods/85cafbe7-aaa6-44b9-80db-9bb290c7a45d/volumes" Jul 09 12:20:08.116910 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.116874 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d998dcf5-z5qr9"] Jul 09 12:20:08.117397 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.117261 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" containerName="manager" Jul 09 12:20:08.117397 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.117273 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" containerName="manager" Jul 09 12:20:08.117397 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.117333 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="85cafbe7-aaa6-44b9-80db-9bb290c7a45d" containerName="manager" Jul 09 12:20:08.141471 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.141439 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d998dcf5-z5qr9"] Jul 09 12:20:08.141611 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.141532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.269841 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.269803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.269851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-service-ca\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.269876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4sh\" (UniqueName: \"kubernetes.io/projected/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-kube-api-access-sh4sh\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.270007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-trusted-ca-bundle\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270073 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.270041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-oauth-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.270075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-oauth-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.270213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.270131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.370832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-trusted-ca-bundle\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.370832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-oauth-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.370832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-oauth-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.370832 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-service-ca\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.370972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4sh\" (UniqueName: \"kubernetes.io/projected/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-kube-api-access-sh4sh\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371694 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.371668 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-oauth-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371823 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.371724 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-service-ca\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371823 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.371724 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.371966 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.371944 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-trusted-ca-bundle\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.373290 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.373266 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-oauth-config\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.373861 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.373841 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-console-serving-cert\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.379898 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.379869 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4sh\" (UniqueName: \"kubernetes.io/projected/1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c-kube-api-access-sh4sh\") pod \"console-67d998dcf5-z5qr9\" (UID: \"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c\") " pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.452009 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.451969 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:08.612127 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.612105 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d998dcf5-z5qr9"] Jul 09 12:20:08.614055 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:08.614023 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6d7696_dc84_45f7_aaa2_9ecdcd546a5c.slice/crio-99e2238504516c66a0c372a95ae3480be3db75bacba84e5f648065dddcd125fb WatchSource:0}: Error finding container 99e2238504516c66a0c372a95ae3480be3db75bacba84e5f648065dddcd125fb: Status 404 returned error can't find the container with id 99e2238504516c66a0c372a95ae3480be3db75bacba84e5f648065dddcd125fb Jul 09 12:20:08.685412 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:08.685372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d998dcf5-z5qr9" event={"ID":"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c","Type":"ContainerStarted","Data":"99e2238504516c66a0c372a95ae3480be3db75bacba84e5f648065dddcd125fb"} Jul 09 12:20:09.692097 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:09.692058 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d998dcf5-z5qr9" event={"ID":"1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c","Type":"ContainerStarted","Data":"85483119612d28bc57d4da33e5858d76a8bd8f0f49d9330f8f0f36f50b68d4a7"} Jul 09 12:20:09.710446 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:09.710398 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d998dcf5-z5qr9" podStartSLOduration=1.710382424 podStartE2EDuration="1.710382424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:09.708286963 +0000 UTC m=+1044.471012510" watchObservedRunningTime="2026-07-09 12:20:09.710382424 +0000 UTC m=+1044.473107960" Jul 09 12:20:18.452604 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:18.452564 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:18.452991 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:18.452629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:18.457325 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:18.457306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:18.739159 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:18.739077 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d998dcf5-z5qr9" Jul 09 12:20:18.796711 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:18.796678 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:20:22.282549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.282514 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:22.285405 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.285390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.287751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.287711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk6c\" (UniqueName: \"kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.287877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.287774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.287877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.287799 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 12:20:22.287877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.287813 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p6dx7\"" Jul 09 12:20:22.295708 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.295682 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:22.386128 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.386090 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:22.388219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.388194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk6c\" (UniqueName: \"kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.388347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.388245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.388881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.388862 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.396918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.396895 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk6c\" (UniqueName: \"kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c\") pod \"limitador-limitador-64c94fd64f-kcnzj\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.597275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.597194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:22.732840 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.732770 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:22.735560 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:22.735525 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe9d3eb_2750_4349_96eb_ecf8aa65afe9.slice/crio-ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2 WatchSource:0}: Error finding container ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2: Status 404 returned error can't find the container with id ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2 Jul 09 12:20:22.755812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:22.755779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" event={"ID":"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9","Type":"ContainerStarted","Data":"ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2"} Jul 09 12:20:23.116056 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.116025 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:23.119232 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.119194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:23.121979 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.121951 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rpqb7\"" Jul 09 12:20:23.150037 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.150012 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:23.195406 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.195372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vsvh\" (UniqueName: \"kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh\") pod \"authorino-58cd955675-p9lc8\" (UID: \"bfdf4819-3621-4e92-ba33-cd805be5d241\") " pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:23.296052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.296014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vsvh\" (UniqueName: \"kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh\") pod \"authorino-58cd955675-p9lc8\" (UID: \"bfdf4819-3621-4e92-ba33-cd805be5d241\") " pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:23.305224 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.305202 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vsvh\" (UniqueName: \"kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh\") pod \"authorino-58cd955675-p9lc8\" (UID: \"bfdf4819-3621-4e92-ba33-cd805be5d241\") " pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:23.326315 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.326288 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:20:23.328845 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.328829 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:23.340992 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.340970 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:20:23.397383 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.397282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9x4\" (UniqueName: \"kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4\") pod \"authorino-75d556bd77-7l8cb\" (UID: \"88e8fca7-d725-4e93-9f34-f35b58f0b2b8\") " pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:23.428735 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.428705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:23.498414 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.498040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9x4\" (UniqueName: \"kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4\") pod \"authorino-75d556bd77-7l8cb\" (UID: \"88e8fca7-d725-4e93-9f34-f35b58f0b2b8\") " pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:23.511481 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.511422 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9x4\" (UniqueName: \"kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4\") pod \"authorino-75d556bd77-7l8cb\" (UID: \"88e8fca7-d725-4e93-9f34-f35b58f0b2b8\") " pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:23.593009 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.592389 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:23.595648 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:23.595618 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfdf4819_3621_4e92_ba33_cd805be5d241.slice/crio-4dcea54e7b72d7b9914273157c89f06fdd9cba7d50d7aa6ccd8c17866b83a7f5 WatchSource:0}: Error finding container 4dcea54e7b72d7b9914273157c89f06fdd9cba7d50d7aa6ccd8c17866b83a7f5: Status 404 returned error can't find the container with id 4dcea54e7b72d7b9914273157c89f06fdd9cba7d50d7aa6ccd8c17866b83a7f5 Jul 09 12:20:23.638803 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.638768 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:23.769658 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.769582 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p9lc8" event={"ID":"bfdf4819-3621-4e92-ba33-cd805be5d241","Type":"ContainerStarted","Data":"4dcea54e7b72d7b9914273157c89f06fdd9cba7d50d7aa6ccd8c17866b83a7f5"} Jul 09 12:20:23.815606 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:23.815491 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:20:23.818482 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:23.818451 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e8fca7_d725_4e93_9f34_f35b58f0b2b8.slice/crio-9501510f26857f738b8f09d45b764321efeca08868c1c088f67c0326828913a0 WatchSource:0}: Error finding container 9501510f26857f738b8f09d45b764321efeca08868c1c088f67c0326828913a0: Status 404 returned error can't find the container with id 9501510f26857f738b8f09d45b764321efeca08868c1c088f67c0326828913a0 Jul 09 12:20:24.780353 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:24.780314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7l8cb" event={"ID":"88e8fca7-d725-4e93-9f34-f35b58f0b2b8","Type":"ContainerStarted","Data":"9501510f26857f738b8f09d45b764321efeca08868c1c088f67c0326828913a0"} Jul 09 12:20:27.798125 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.798089 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" event={"ID":"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9","Type":"ContainerStarted","Data":"52f41c19ab318444d2a6444d0d7991419f24bd16e70d2d2cf1ec64295792360d"} Jul 09 12:20:27.798563 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.798153 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:27.799574 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.799547 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p9lc8" event={"ID":"bfdf4819-3621-4e92-ba33-cd805be5d241","Type":"ContainerStarted","Data":"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e"} Jul 09 12:20:27.800735 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.800710 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7l8cb" event={"ID":"88e8fca7-d725-4e93-9f34-f35b58f0b2b8","Type":"ContainerStarted","Data":"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0"} Jul 09 12:20:27.817167 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.817126 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" podStartSLOduration=1.211237954 podStartE2EDuration="5.817112567s" totalImagesPullingTime="4.605874613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:27.815792385 +0000 UTC m=+1062.578517926" watchObservedRunningTime="2026-07-09 12:20:27.817112567 +0000 UTC m=+1062.579838101" Jul 09 12:20:27.833863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.833822 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-7l8cb" podStartSLOduration=1.238666794 podStartE2EDuration="4.833811336s" totalImagesPullingTime="3.595144542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:27.831799317 +0000 UTC m=+1062.594524851" watchObservedRunningTime="2026-07-09 12:20:27.833811336 +0000 UTC m=+1062.596536870" Jul 09 12:20:27.846472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.846422 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-p9lc8" podStartSLOduration=1.073069324 podStartE2EDuration="4.846405397s" totalImagesPullingTime="3.773336073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:27.845219028 +0000 UTC m=+1062.607944564" watchObservedRunningTime="2026-07-09 12:20:27.846405397 +0000 UTC m=+1062.609130934" Jul 09 12:20:27.869471 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:27.869444 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:29.810464 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:29.810427 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-p9lc8" podUID="bfdf4819-3621-4e92-ba33-cd805be5d241" containerName="authorino" containerID="cri-o://9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e" gracePeriod=30 Jul 09 12:20:30.068477 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.068422 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:30.162874 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.162837 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vsvh\" (UniqueName: \"kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh\") pod \"bfdf4819-3621-4e92-ba33-cd805be5d241\" (UID: \"bfdf4819-3621-4e92-ba33-cd805be5d241\") " Jul 09 12:20:30.165159 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.165125 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh" (OuterVolumeSpecName: "kube-api-access-9vsvh") pod "bfdf4819-3621-4e92-ba33-cd805be5d241" (UID: "bfdf4819-3621-4e92-ba33-cd805be5d241"). InnerVolumeSpecName "kube-api-access-9vsvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:30.264392 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.264362 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vsvh\" (UniqueName: \"kubernetes.io/projected/bfdf4819-3621-4e92-ba33-cd805be5d241-kube-api-access-9vsvh\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:30.816880 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.816851 2567 generic.go:356] "Generic (PLEG): container finished" podID="bfdf4819-3621-4e92-ba33-cd805be5d241" containerID="9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e" exitCode=0 Jul 09 12:20:30.817266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.816896 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p9lc8" Jul 09 12:20:30.817266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.816898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p9lc8" event={"ID":"bfdf4819-3621-4e92-ba33-cd805be5d241","Type":"ContainerDied","Data":"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e"} Jul 09 12:20:30.817266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.817022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p9lc8" event={"ID":"bfdf4819-3621-4e92-ba33-cd805be5d241","Type":"ContainerDied","Data":"4dcea54e7b72d7b9914273157c89f06fdd9cba7d50d7aa6ccd8c17866b83a7f5"} Jul 09 12:20:30.817266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.817048 2567 scope.go:117] "RemoveContainer" containerID="9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e" Jul 09 12:20:30.825775 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.825759 2567 scope.go:117] "RemoveContainer" containerID="9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e" Jul 09 12:20:30.826068 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:20:30.826047 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e\": container with ID starting with 9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e not found: ID does not exist" containerID="9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e" Jul 09 12:20:30.826151 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.826077 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e"} err="failed to get container status \"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e\": rpc error: code = NotFound desc = could not find container \"9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e\": container with ID starting with 9fd20f3894f072a594c6c58f25af8ae8448d9b608068a82875cad1d5c3f90f7e not found: ID does not exist" Jul 09 12:20:30.836831 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.836808 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:30.841028 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:30.841009 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-p9lc8"] Jul 09 12:20:31.847287 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:31.847255 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdf4819-3621-4e92-ba33-cd805be5d241" path="/var/lib/kubelet/pods/bfdf4819-3621-4e92-ba33-cd805be5d241/volumes" Jul 09 12:20:38.318457 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.318375 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:38.318901 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.318599 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" podUID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" containerName="limitador" containerID="cri-o://52f41c19ab318444d2a6444d0d7991419f24bd16e70d2d2cf1ec64295792360d" gracePeriod=30 Jul 09 12:20:38.319489 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.319467 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:38.859380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.859348 2567 generic.go:356] "Generic (PLEG): container finished" podID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" containerID="52f41c19ab318444d2a6444d0d7991419f24bd16e70d2d2cf1ec64295792360d" exitCode=0 Jul 09 12:20:38.859490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.859425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" event={"ID":"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9","Type":"ContainerDied","Data":"52f41c19ab318444d2a6444d0d7991419f24bd16e70d2d2cf1ec64295792360d"} Jul 09 12:20:38.859490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.859471 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" event={"ID":"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9","Type":"ContainerDied","Data":"ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2"} Jul 09 12:20:38.859559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.859496 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef0f277e69701a908da7729f91d9d30f9606a5eb7125943c3cef1d371dd9b8c2" Jul 09 12:20:38.866490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.866466 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:38.942656 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.942619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tk6c\" (UniqueName: \"kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c\") pod \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " Jul 09 12:20:38.942656 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.942663 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file\") pod \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\" (UID: \"fbe9d3eb-2750-4349-96eb-ecf8aa65afe9\") " Jul 09 12:20:38.943127 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.943097 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file" (OuterVolumeSpecName: "config-file") pod "fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" (UID: "fbe9d3eb-2750-4349-96eb-ecf8aa65afe9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:20:38.944853 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:38.944833 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c" (OuterVolumeSpecName: "kube-api-access-5tk6c") pod "fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" (UID: "fbe9d3eb-2750-4349-96eb-ecf8aa65afe9"). InnerVolumeSpecName "kube-api-access-5tk6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:39.044188 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:39.044131 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk6c\" (UniqueName: \"kubernetes.io/projected/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-kube-api-access-5tk6c\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:39.044188 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:39.044180 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9-config-file\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:39.863768 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:39.863743 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kcnzj" Jul 09 12:20:39.879637 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:39.879609 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:39.883042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:39.883024 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kcnzj"] Jul 09 12:20:41.848158 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:41.848127 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" path="/var/lib/kubelet/pods/fbe9d3eb-2750-4349-96eb-ecf8aa65afe9/volumes" Jul 09 12:20:43.819493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.819452 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d995c7998-spnpc" podUID="1a847330-acf4-41c3-9909-04bd8b6021cc" containerName="console" containerID="cri-o://e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9" gracePeriod=15 Jul 09 12:20:43.848677 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.848651 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-swdvz"] Jul 09 12:20:43.849103 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849086 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdf4819-3621-4e92-ba33-cd805be5d241" containerName="authorino" Jul 09 12:20:43.849103 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849103 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdf4819-3621-4e92-ba33-cd805be5d241" containerName="authorino" Jul 09 12:20:43.849254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849117 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" containerName="limitador" Jul 09 12:20:43.849254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849124 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" containerName="limitador" Jul 09 12:20:43.849254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849222 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe9d3eb-2750-4349-96eb-ecf8aa65afe9" containerName="limitador" Jul 09 12:20:43.849254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.849238 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdf4819-3621-4e92-ba33-cd805be5d241" containerName="authorino" Jul 09 12:20:43.853497 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.853470 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:43.856612 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.856587 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 12:20:43.856767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.856745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qk8ng\"" Jul 09 12:20:43.857307 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.857287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 12:20:43.857605 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.857318 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 12:20:43.858479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.858457 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-swdvz"] Jul 09 12:20:43.990621 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.990578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ccfa6e4-19a4-4544-ad1e-c50dad850989-data\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:43.990765 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:43.990643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzkx\" (UniqueName: \"kubernetes.io/projected/9ccfa6e4-19a4-4544-ad1e-c50dad850989-kube-api-access-qkzkx\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.071042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.070989 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:20:44.091385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.091357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ccfa6e4-19a4-4544-ad1e-c50dad850989-data\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.091526 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.091420 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzkx\" (UniqueName: \"kubernetes.io/projected/9ccfa6e4-19a4-4544-ad1e-c50dad850989-kube-api-access-qkzkx\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.091784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.091762 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ccfa6e4-19a4-4544-ad1e-c50dad850989-data\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.102110 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.102089 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzkx\" (UniqueName: \"kubernetes.io/projected/9ccfa6e4-19a4-4544-ad1e-c50dad850989-kube-api-access-qkzkx\") pod \"postgres-69fb4594cf-swdvz\" (UID: \"9ccfa6e4-19a4-4544-ad1e-c50dad850989\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.183472 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.183452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:44.192534 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192516 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192569 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pslz7\" (UniqueName: \"kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192628 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192653 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192684 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192745 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config\") pod \"1a847330-acf4-41c3-9909-04bd8b6021cc\" (UID: \"1a847330-acf4-41c3-9909-04bd8b6021cc\") " Jul 09 12:20:44.192991 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.192967 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:20:44.193104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.193084 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-oauth-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.193353 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.193311 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:20:44.193353 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.193223 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:20:44.193515 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.193423 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config" (OuterVolumeSpecName: "console-config") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:20:44.195214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.195189 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:20:44.195320 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.195304 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7" (OuterVolumeSpecName: "kube-api-access-pslz7") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "kube-api-access-pslz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:44.195603 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.195574 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a847330-acf4-41c3-9909-04bd8b6021cc" (UID: "1a847330-acf4-41c3-9909-04bd8b6021cc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:20:44.294619 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294583 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslz7\" (UniqueName: \"kubernetes.io/projected/1a847330-acf4-41c3-9909-04bd8b6021cc-kube-api-access-pslz7\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.294619 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294618 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-oauth-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.294849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294628 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a847330-acf4-41c3-9909-04bd8b6021cc-console-serving-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.294849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294637 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-service-ca\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.294849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294645 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-trusted-ca-bundle\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.294849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.294653 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a847330-acf4-41c3-9909-04bd8b6021cc-console-config\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:44.315576 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.315554 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-swdvz"] Jul 09 12:20:44.317570 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:44.317541 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccfa6e4_19a4_4544_ad1e_c50dad850989.slice/crio-bf77b75e0f5cea251968bd910ae49578687cec0434ba5f9ae9875d53253872a4 WatchSource:0}: Error finding container bf77b75e0f5cea251968bd910ae49578687cec0434ba5f9ae9875d53253872a4: Status 404 returned error can't find the container with id bf77b75e0f5cea251968bd910ae49578687cec0434ba5f9ae9875d53253872a4 Jul 09 12:20:44.890018 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.889982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" event={"ID":"9ccfa6e4-19a4-4544-ad1e-c50dad850989","Type":"ContainerStarted","Data":"bf77b75e0f5cea251968bd910ae49578687cec0434ba5f9ae9875d53253872a4"} Jul 09 12:20:44.891307 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.891284 2567 generic.go:356] "Generic (PLEG): container finished" podID="1a847330-acf4-41c3-9909-04bd8b6021cc" containerID="e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9" exitCode=0 Jul 09 12:20:44.891451 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.891343 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d995c7998-spnpc" Jul 09 12:20:44.891451 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.891369 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d995c7998-spnpc" event={"ID":"1a847330-acf4-41c3-9909-04bd8b6021cc","Type":"ContainerDied","Data":"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9"} Jul 09 12:20:44.891451 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.891414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d995c7998-spnpc" event={"ID":"1a847330-acf4-41c3-9909-04bd8b6021cc","Type":"ContainerDied","Data":"f633d7ebcf2c699074226961991f34e3c9da87c7846c72247aef1d8dfcc26c9f"} Jul 09 12:20:44.891451 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.891435 2567 scope.go:117] "RemoveContainer" containerID="e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9" Jul 09 12:20:44.900981 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.900960 2567 scope.go:117] "RemoveContainer" containerID="e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9" Jul 09 12:20:44.901225 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:20:44.901206 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9\": container with ID starting with e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9 not found: ID does not exist" containerID="e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9" Jul 09 12:20:44.901292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.901234 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9"} err="failed to get container status \"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9\": rpc error: code = NotFound desc = could not find container \"e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9\": container with ID starting with e5b25f74b211d100c289e0d08f8cd33c2f2567aa9ef64f900f7552900e3879e9 not found: ID does not exist" Jul 09 12:20:44.912426 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.912405 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:20:44.916773 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:44.916754 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d995c7998-spnpc"] Jul 09 12:20:45.852771 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:45.852737 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a847330-acf4-41c3-9909-04bd8b6021cc" path="/var/lib/kubelet/pods/1a847330-acf4-41c3-9909-04bd8b6021cc/volumes" Jul 09 12:20:49.305069 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:49.305046 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 12:20:49.921836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:49.921803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" event={"ID":"9ccfa6e4-19a4-4544-ad1e-c50dad850989","Type":"ContainerStarted","Data":"37a7d3bc1e2a4706d1cedba155c88e3b2f0ab6830eba69d36916b4f19b077640"} Jul 09 12:20:49.922044 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:49.921916 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:49.939080 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:49.939031 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" podStartSLOduration=1.9555275239999999 podStartE2EDuration="6.939017256s" totalImagesPullingTime="4.983489732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:49.937893939 +0000 UTC m=+1084.700619457" watchObservedRunningTime="2026-07-09 12:20:49.939017256 +0000 UTC m=+1084.701742801" Jul 09 12:20:55.960034 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:55.960002 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-swdvz" Jul 09 12:20:56.489818 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.489779 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwv6c"] Jul 09 12:20:56.490395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.490377 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a847330-acf4-41c3-9909-04bd8b6021cc" containerName="console" Jul 09 12:20:56.490456 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.490398 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a847330-acf4-41c3-9909-04bd8b6021cc" containerName="console" Jul 09 12:20:56.490498 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.490489 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a847330-acf4-41c3-9909-04bd8b6021cc" containerName="console" Jul 09 12:20:56.570991 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.570923 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwv6c"] Jul 09 12:20:56.571137 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.571043 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:56.709324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.709290 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnk5\" (UniqueName: \"kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5\") pod \"authorino-5d4b67c446-wwv6c\" (UID: \"2ce398f8-0001-484f-a595-fc887eba0382\") " pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:56.715127 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.715104 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwv6c"] Jul 09 12:20:56.715346 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:20:56.715326 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kcnk5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-wwv6c" podUID="2ce398f8-0001-484f-a595-fc887eba0382" Jul 09 12:20:56.810033 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.809954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnk5\" (UniqueName: \"kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5\") pod \"authorino-5d4b67c446-wwv6c\" (UID: \"2ce398f8-0001-484f-a595-fc887eba0382\") " pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:56.819100 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.819074 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnk5\" (UniqueName: \"kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5\") pod \"authorino-5d4b67c446-wwv6c\" (UID: \"2ce398f8-0001-484f-a595-fc887eba0382\") " pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:56.873558 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.873520 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:20:56.907863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.907833 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:20:56.908052 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.907922 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:56.910380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.910357 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 12:20:56.953916 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.953877 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:56.984033 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:56.984010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:57.011412 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.011383 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkcwp\" (UniqueName: \"kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.011556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.011436 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.112348 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.112257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcnk5\" (UniqueName: \"kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5\") pod \"2ce398f8-0001-484f-a595-fc887eba0382\" (UID: \"2ce398f8-0001-484f-a595-fc887eba0382\") " Jul 09 12:20:57.112501 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.112441 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkcwp\" (UniqueName: \"kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.112561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.112535 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.114610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.114584 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5" (OuterVolumeSpecName: "kube-api-access-kcnk5") pod "2ce398f8-0001-484f-a595-fc887eba0382" (UID: "2ce398f8-0001-484f-a595-fc887eba0382"). InnerVolumeSpecName "kube-api-access-kcnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:57.115234 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.115213 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.120222 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.120198 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkcwp\" (UniqueName: \"kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp\") pod \"authorino-b6c7fff-z4kl5\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.213618 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.213583 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnk5\" (UniqueName: \"kubernetes.io/projected/2ce398f8-0001-484f-a595-fc887eba0382-kube-api-access-kcnk5\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:57.217373 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.217349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:20:57.373458 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.373434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:20:57.375598 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:20:57.375568 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod457671cd_0220_433e_91e4_d88c27a66e9e.slice/crio-d7671946414f2b8be99e134da411bd371be12dc1073d72678f23ac0488a29427 WatchSource:0}: Error finding container d7671946414f2b8be99e134da411bd371be12dc1073d72678f23ac0488a29427: Status 404 returned error can't find the container with id d7671946414f2b8be99e134da411bd371be12dc1073d72678f23ac0488a29427 Jul 09 12:20:57.959277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.959244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c7fff-z4kl5" event={"ID":"457671cd-0220-433e-91e4-d88c27a66e9e","Type":"ContainerStarted","Data":"d7671946414f2b8be99e134da411bd371be12dc1073d72678f23ac0488a29427"} Jul 09 12:20:57.959277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.959265 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwv6c" Jul 09 12:20:57.991144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.991114 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwv6c"] Jul 09 12:20:57.994524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:57.994496 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwv6c"] Jul 09 12:20:58.965381 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:58.965342 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c7fff-z4kl5" event={"ID":"457671cd-0220-433e-91e4-d88c27a66e9e","Type":"ContainerStarted","Data":"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213"} Jul 09 12:20:58.982263 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:58.982214 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c7fff-z4kl5" podStartSLOduration=2.130414169 podStartE2EDuration="2.982198048s" totalImagesPullingTime="851.783879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:20:58.980622443 +0000 UTC m=+1093.743347981" watchObservedRunningTime="2026-07-09 12:20:58.982198048 +0000 UTC m=+1093.744923593" Jul 09 12:20:59.006359 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.006333 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:20:59.006794 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.006507 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-7l8cb" podUID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" containerName="authorino" containerID="cri-o://19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0" gracePeriod=30 Jul 09 12:20:59.259099 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.259072 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:59.433387 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.433348 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn9x4\" (UniqueName: \"kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4\") pod \"88e8fca7-d725-4e93-9f34-f35b58f0b2b8\" (UID: \"88e8fca7-d725-4e93-9f34-f35b58f0b2b8\") " Jul 09 12:20:59.435709 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.435682 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4" (OuterVolumeSpecName: "kube-api-access-gn9x4") pod "88e8fca7-d725-4e93-9f34-f35b58f0b2b8" (UID: "88e8fca7-d725-4e93-9f34-f35b58f0b2b8"). InnerVolumeSpecName "kube-api-access-gn9x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:20:59.534418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.534314 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9x4\" (UniqueName: \"kubernetes.io/projected/88e8fca7-d725-4e93-9f34-f35b58f0b2b8-kube-api-access-gn9x4\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:20:59.848627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.848547 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce398f8-0001-484f-a595-fc887eba0382" path="/var/lib/kubelet/pods/2ce398f8-0001-484f-a595-fc887eba0382/volumes" Jul 09 12:20:59.971253 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.971220 2567 generic.go:356] "Generic (PLEG): container finished" podID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" containerID="19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0" exitCode=0 Jul 09 12:20:59.971408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.971262 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7l8cb" Jul 09 12:20:59.971408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.971306 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7l8cb" event={"ID":"88e8fca7-d725-4e93-9f34-f35b58f0b2b8","Type":"ContainerDied","Data":"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0"} Jul 09 12:20:59.971408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.971348 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7l8cb" event={"ID":"88e8fca7-d725-4e93-9f34-f35b58f0b2b8","Type":"ContainerDied","Data":"9501510f26857f738b8f09d45b764321efeca08868c1c088f67c0326828913a0"} Jul 09 12:20:59.971408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.971365 2567 scope.go:117] "RemoveContainer" containerID="19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0" Jul 09 12:20:59.979999 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.979980 2567 scope.go:117] "RemoveContainer" containerID="19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0" Jul 09 12:20:59.980263 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:20:59.980244 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0\": container with ID starting with 19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0 not found: ID does not exist" containerID="19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0" Jul 09 12:20:59.980312 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.980271 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0"} err="failed to get container status \"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0\": rpc error: code = NotFound desc = could not find container \"19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0\": container with ID starting with 19ab00684ce70233538f9a605cc740a96a7712620bd8b24de44f276da8a140b0 not found: ID does not exist" Jul 09 12:20:59.987484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.987464 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:20:59.991005 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:20:59.990985 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7l8cb"] Jul 09 12:21:01.853228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:01.847914 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" path="/var/lib/kubelet/pods/88e8fca7-d725-4e93-9f34-f35b58f0b2b8/volumes" Jul 09 12:21:03.170426 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.170395 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:03.170822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.170808 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" containerName="authorino" Jul 09 12:21:03.170866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.170824 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" containerName="authorino" Jul 09 12:21:03.170900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.170895 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e8fca7-d725-4e93-9f34-f35b58f0b2b8" containerName="authorino" Jul 09 12:21:03.174320 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.174299 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.176724 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.176705 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dwnjj\"" Jul 09 12:21:03.176845 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.176830 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 12:21:03.177349 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.177333 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 12:21:03.185613 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.185590 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:03.348457 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.348425 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:21:03.351150 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.351127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.362707 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.362679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:21:03.369102 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.369077 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.369235 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.369126 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtbf\" (UniqueName: \"kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.469875 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.469842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtbf\" (UniqueName: \"kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.470129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.469916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbh7\" (UniqueName: \"kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.470129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.470035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.470129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.470085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.470259 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:21:03.470184 2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 12:21:03.470259 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:21:03.470244 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert podName:e8cfe32a-0774-4574-9e25-646378a689ab nodeName:}" failed. No retries permitted until 2026-07-09 12:21:03.970226026 +0000 UTC m=+1098.732951541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert") pod "maas-controller-69c46945f9-ztcsq" (UID: "e8cfe32a-0774-4574-9e25-646378a689ab") : secret "maas-controller-webhook-cert" not found Jul 09 12:21:03.480607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.480575 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtbf\" (UniqueName: \"kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.571172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.571131 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbh7\" (UniqueName: \"kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.571343 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.571217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.571384 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:21:03.571343 2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 12:21:03.571419 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:21:03.571389 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert podName:74b703b6-6544-4edb-9d3f-33738d7f5053 nodeName:}" failed. No retries permitted until 2026-07-09 12:21:04.071374902 +0000 UTC m=+1098.834100416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert") pod "maas-controller-79f8c776fc-mpvb7" (UID: "74b703b6-6544-4edb-9d3f-33738d7f5053") : secret "maas-controller-webhook-cert" not found Jul 09 12:21:03.579627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.579593 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbh7\" (UniqueName: \"kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:03.976223 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.976187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:03.978725 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:03.978706 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") pod \"maas-controller-69c46945f9-ztcsq\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:04.076711 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.076670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:04.079239 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.079218 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") pod \"maas-controller-79f8c776fc-mpvb7\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:04.086224 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.086200 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:04.219338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.219312 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:04.221725 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:21:04.221702 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cfe32a_0774_4574_9e25_646378a689ab.slice/crio-bc9882efd5844dcf34d8c6804be04e921f19cfecd30b5b169dd78f121f99e94b WatchSource:0}: Error finding container bc9882efd5844dcf34d8c6804be04e921f19cfecd30b5b169dd78f121f99e94b: Status 404 returned error can't find the container with id bc9882efd5844dcf34d8c6804be04e921f19cfecd30b5b169dd78f121f99e94b Jul 09 12:21:04.264781 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.264721 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:04.398395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:04.398365 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:21:04.400782 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:21:04.400748 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b703b6_6544_4edb_9d3f_33738d7f5053.slice/crio-5ccfabddef6b05ceccc3c6cab9136ec8302da56299aacfed9b7045baecc9b90e WatchSource:0}: Error finding container 5ccfabddef6b05ceccc3c6cab9136ec8302da56299aacfed9b7045baecc9b90e: Status 404 returned error can't find the container with id 5ccfabddef6b05ceccc3c6cab9136ec8302da56299aacfed9b7045baecc9b90e Jul 09 12:21:05.003838 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:05.003791 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c46945f9-ztcsq" event={"ID":"e8cfe32a-0774-4574-9e25-646378a689ab","Type":"ContainerStarted","Data":"bc9882efd5844dcf34d8c6804be04e921f19cfecd30b5b169dd78f121f99e94b"} Jul 09 12:21:05.006284 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:05.006192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" event={"ID":"74b703b6-6544-4edb-9d3f-33738d7f5053","Type":"ContainerStarted","Data":"5ccfabddef6b05ceccc3c6cab9136ec8302da56299aacfed9b7045baecc9b90e"} Jul 09 12:21:08.023457 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.023419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c46945f9-ztcsq" event={"ID":"e8cfe32a-0774-4574-9e25-646378a689ab","Type":"ContainerStarted","Data":"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da"} Jul 09 12:21:08.023888 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.023604 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:08.024913 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.024889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" event={"ID":"74b703b6-6544-4edb-9d3f-33738d7f5053","Type":"ContainerStarted","Data":"0d274c7e220fdbd31d1a48d164e413af1220ee9715ed67f4b4c045641c676472"} Jul 09 12:21:08.025050 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.024969 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:08.050168 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.050119 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69c46945f9-ztcsq" podStartSLOduration=2.244810404 podStartE2EDuration="5.050104099s" totalImagesPullingTime="2.805293695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:21:08.048342327 +0000 UTC m=+1102.811067862" watchObservedRunningTime="2026-07-09 12:21:08.050104099 +0000 UTC m=+1102.812829634" Jul 09 12:21:08.070309 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:08.070256 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" podStartSLOduration=2.454704453 podStartE2EDuration="5.070238843s" totalImagesPullingTime="2.61553439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:21:08.069139056 +0000 UTC m=+1102.831864612" watchObservedRunningTime="2026-07-09 12:21:08.070238843 +0000 UTC m=+1102.832964379" Jul 09 12:21:19.038250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.038217 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:21:19.038655 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.038290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:19.096089 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.096053 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:19.096332 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.096294 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-69c46945f9-ztcsq" podUID="e8cfe32a-0774-4574-9e25-646378a689ab" containerName="manager" containerID="cri-o://13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da" gracePeriod=30 Jul 09 12:21:19.355868 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.355840 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:19.407270 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.407238 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") pod \"e8cfe32a-0774-4574-9e25-646378a689ab\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " Jul 09 12:21:19.407439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.407402 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrtbf\" (UniqueName: \"kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf\") pod \"e8cfe32a-0774-4574-9e25-646378a689ab\" (UID: \"e8cfe32a-0774-4574-9e25-646378a689ab\") " Jul 09 12:21:19.409568 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.409540 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf" (OuterVolumeSpecName: "kube-api-access-qrtbf") pod "e8cfe32a-0774-4574-9e25-646378a689ab" (UID: "e8cfe32a-0774-4574-9e25-646378a689ab"). InnerVolumeSpecName "kube-api-access-qrtbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:21:19.409671 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.409589 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e8cfe32a-0774-4574-9e25-646378a689ab" (UID: "e8cfe32a-0774-4574-9e25-646378a689ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:21:19.508302 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.508273 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrtbf\" (UniqueName: \"kubernetes.io/projected/e8cfe32a-0774-4574-9e25-646378a689ab-kube-api-access-qrtbf\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:21:19.508302 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:19.508299 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8cfe32a-0774-4574-9e25-646378a689ab-webhook-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:21:20.088731 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.088696 2567 generic.go:356] "Generic (PLEG): container finished" podID="e8cfe32a-0774-4574-9e25-646378a689ab" containerID="13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da" exitCode=0 Jul 09 12:21:20.089212 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.088740 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c46945f9-ztcsq" Jul 09 12:21:20.089212 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.088783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c46945f9-ztcsq" event={"ID":"e8cfe32a-0774-4574-9e25-646378a689ab","Type":"ContainerDied","Data":"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da"} Jul 09 12:21:20.089212 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.088833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c46945f9-ztcsq" event={"ID":"e8cfe32a-0774-4574-9e25-646378a689ab","Type":"ContainerDied","Data":"bc9882efd5844dcf34d8c6804be04e921f19cfecd30b5b169dd78f121f99e94b"} Jul 09 12:21:20.089212 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.088856 2567 scope.go:117] "RemoveContainer" containerID="13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da" Jul 09 12:21:20.098103 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.097917 2567 scope.go:117] "RemoveContainer" containerID="13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da" Jul 09 12:21:20.098245 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:21:20.098227 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da\": container with ID starting with 13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da not found: ID does not exist" containerID="13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da" Jul 09 12:21:20.098289 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.098254 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da"} err="failed to get container status \"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da\": rpc error: code = NotFound desc = could not find container \"13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da\": container with ID starting with 13198fa56d1cd97242aaa93bf0c3c93a54761768bc26ff7261af0c6337e414da not found: ID does not exist" Jul 09 12:21:20.104867 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.104811 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:20.107495 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:20.107473 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69c46945f9-ztcsq"] Jul 09 12:21:21.847759 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:21.847725 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cfe32a-0774-4574-9e25-646378a689ab" path="/var/lib/kubelet/pods/e8cfe32a-0774-4574-9e25-646378a689ab/volumes" Jul 09 12:21:39.751735 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.751700 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-r9j7f"] Jul 09 12:21:39.752324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.752304 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cfe32a-0774-4574-9e25-646378a689ab" containerName="manager" Jul 09 12:21:39.752415 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.752329 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cfe32a-0774-4574-9e25-646378a689ab" containerName="manager" Jul 09 12:21:39.752470 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.752416 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cfe32a-0774-4574-9e25-646378a689ab" containerName="manager" Jul 09 12:21:39.775192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.775150 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-r9j7f"] Jul 09 12:21:39.775360 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.775281 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:39.777995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.777970 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 12:21:39.778129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.778068 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9mxc9\"" Jul 09 12:21:39.799298 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.799276 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fkxp4"] Jul 09 12:21:39.802856 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.802840 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.811850 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.811825 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fkxp4"] Jul 09 12:21:39.893751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.893722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae099e53-b928-4451-a8cf-5e03ede71b45-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.893751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.893759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722ee86a-4466-4618-90d1-1963090bf05e-plugins-config-volume\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:39.893988 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.893859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvxct\" (UniqueName: \"kubernetes.io/projected/722ee86a-4466-4618-90d1-1963090bf05e-kube-api-access-tvxct\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:39.893988 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.893916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jbj\" (UniqueName: \"kubernetes.io/projected/ae099e53-b928-4451-a8cf-5e03ede71b45-kube-api-access-j2jbj\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.995151 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.995115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae099e53-b928-4451-a8cf-5e03ede71b45-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.995342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.995171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722ee86a-4466-4618-90d1-1963090bf05e-plugins-config-volume\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:39.995342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.995243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvxct\" (UniqueName: \"kubernetes.io/projected/722ee86a-4466-4618-90d1-1963090bf05e-kube-api-access-tvxct\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:39.995342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.995279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jbj\" (UniqueName: \"kubernetes.io/projected/ae099e53-b928-4451-a8cf-5e03ede71b45-kube-api-access-j2jbj\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.996015 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.995989 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae099e53-b928-4451-a8cf-5e03ede71b45-plugins-config-volume\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:39.996131 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:39.996050 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722ee86a-4466-4618-90d1-1963090bf05e-plugins-config-volume\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:40.005210 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.005153 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jbj\" (UniqueName: \"kubernetes.io/projected/ae099e53-b928-4451-a8cf-5e03ede71b45-kube-api-access-j2jbj\") pod \"payload-pre-processing-7964457696-fkxp4\" (UID: \"ae099e53-b928-4451-a8cf-5e03ede71b45\") " pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:40.005313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.005210 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvxct\" (UniqueName: \"kubernetes.io/projected/722ee86a-4466-4618-90d1-1963090bf05e-kube-api-access-tvxct\") pod \"payload-processing-58f86c7798-r9j7f\" (UID: \"722ee86a-4466-4618-90d1-1963090bf05e\") " pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:40.087919 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.087883 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:40.113344 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.113313 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:40.247976 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.247899 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-r9j7f"] Jul 09 12:21:40.251878 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:21:40.251850 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722ee86a_4466_4618_90d1_1963090bf05e.slice/crio-56db1f612314160b2ce58c3d284caa786f83dca7e65be47f7ab0910bd6d09855 WatchSource:0}: Error finding container 56db1f612314160b2ce58c3d284caa786f83dca7e65be47f7ab0910bd6d09855: Status 404 returned error can't find the container with id 56db1f612314160b2ce58c3d284caa786f83dca7e65be47f7ab0910bd6d09855 Jul 09 12:21:40.272366 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:40.272339 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-fkxp4"] Jul 09 12:21:40.274299 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:21:40.274274 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae099e53_b928_4451_a8cf_5e03ede71b45.slice/crio-e532442ba4ef5d2b65c4e632e287abed83780c513ecbdfbd317982ae91ca376b WatchSource:0}: Error finding container e532442ba4ef5d2b65c4e632e287abed83780c513ecbdfbd317982ae91ca376b: Status 404 returned error can't find the container with id e532442ba4ef5d2b65c4e632e287abed83780c513ecbdfbd317982ae91ca376b Jul 09 12:21:41.202118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:41.202041 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" event={"ID":"722ee86a-4466-4618-90d1-1963090bf05e","Type":"ContainerStarted","Data":"56db1f612314160b2ce58c3d284caa786f83dca7e65be47f7ab0910bd6d09855"} Jul 09 12:21:41.203675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:41.203645 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" event={"ID":"ae099e53-b928-4451-a8cf-5e03ede71b45","Type":"ContainerStarted","Data":"e532442ba4ef5d2b65c4e632e287abed83780c513ecbdfbd317982ae91ca376b"} Jul 09 12:21:42.211159 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.211122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" event={"ID":"ae099e53-b928-4451-a8cf-5e03ede71b45","Type":"ContainerStarted","Data":"ccc90afeee11ab84ef0aafebd6b5a9f35e41de8e04007a22a44fac92f016b682"} Jul 09 12:21:42.211568 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.211193 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:21:42.212814 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.212789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" event={"ID":"722ee86a-4466-4618-90d1-1963090bf05e","Type":"ContainerStarted","Data":"b1ade0f4fb7f62af78450b83eba22301ba6c9ae487883aa586e7839a705cdd59"} Jul 09 12:21:42.212995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.212915 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:42.228277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.228221 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" podStartSLOduration=1.7789397930000002 podStartE2EDuration="3.228200249s" totalImagesPullingTime="1.449260456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:21:42.225513975 +0000 UTC m=+1136.988239522" watchObservedRunningTime="2026-07-09 12:21:42.228200249 +0000 UTC m=+1136.990925797" Jul 09 12:21:42.243062 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:42.243010 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" podStartSLOduration=1.768892572 podStartE2EDuration="3.242993262s" totalImagesPullingTime="1.47410069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:21:42.241242918 +0000 UTC m=+1137.003968480" watchObservedRunningTime="2026-07-09 12:21:42.242993262 +0000 UTC m=+1137.005718798" Jul 09 12:21:53.219108 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:53.219078 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-r9j7f" Jul 09 12:21:53.219597 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:21:53.219274 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-fkxp4" Jul 09 12:22:03.234493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.234459 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg"] Jul 09 12:22:03.239587 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.239567 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.242141 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.242118 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 12:22:03.243003 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.242979 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sgjkl\"" Jul 09 12:22:03.243003 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.242991 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 12:22:03.243189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.243003 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 12:22:03.247156 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.247131 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg"] Jul 09 12:22:03.295610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.295796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.295796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.295796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295757 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.295796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a99598-b1ec-415b-90c0-3403271088d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.295990 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.295827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tft9m\" (UniqueName: \"kubernetes.io/projected/39a99598-b1ec-415b-90c0-3403271088d1-kube-api-access-tft9m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397179 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397255 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a99598-b1ec-415b-90c0-3403271088d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tft9m\" (UniqueName: \"kubernetes.io/projected/39a99598-b1ec-415b-90c0-3403271088d1-kube-api-access-tft9m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397439 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397739 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.397918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.397875 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.398114 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.398093 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.399594 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.399566 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a99598-b1ec-415b-90c0-3403271088d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.399986 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.399969 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a99598-b1ec-415b-90c0-3403271088d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.405236 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.405214 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tft9m\" (UniqueName: \"kubernetes.io/projected/39a99598-b1ec-415b-90c0-3403271088d1-kube-api-access-tft9m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg\" (UID: \"39a99598-b1ec-415b-90c0-3403271088d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.552622 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.552515 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:03.696219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:03.696147 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg"] Jul 09 12:22:03.699511 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:22:03.699479 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a99598_b1ec_415b_90c0_3403271088d1.slice/crio-2931e4bd6e16dcac9dc61e4d1cc7a3cb0cbcb9825def858862da6d4e7f32edea WatchSource:0}: Error finding container 2931e4bd6e16dcac9dc61e4d1cc7a3cb0cbcb9825def858862da6d4e7f32edea: Status 404 returned error can't find the container with id 2931e4bd6e16dcac9dc61e4d1cc7a3cb0cbcb9825def858862da6d4e7f32edea Jul 09 12:22:04.328464 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:04.328241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" event={"ID":"39a99598-b1ec-415b-90c0-3403271088d1","Type":"ContainerStarted","Data":"2931e4bd6e16dcac9dc61e4d1cc7a3cb0cbcb9825def858862da6d4e7f32edea"} Jul 09 12:22:05.334770 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:05.334682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" event={"ID":"39a99598-b1ec-415b-90c0-3403271088d1","Type":"ContainerStarted","Data":"f5724c25bc51ebf37d9547869d2a4b62504de096d92ccb605633338146daedfb"} Jul 09 12:22:05.353416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:05.353364 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" podStartSLOduration=1.017042618 podStartE2EDuration="2.353349288s" totalImagesPullingTime="1.33630667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:22:05.351061851 +0000 UTC m=+1160.113787378" watchObservedRunningTime="2026-07-09 12:22:05.353349288 +0000 UTC m=+1160.116074860" Jul 09 12:22:08.616992 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.616956 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6v745"] Jul 09 12:22:08.621063 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.621038 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.623543 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.623501 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 12:22:08.631606 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.631579 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6v745"] Jul 09 12:22:08.647666 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.647666 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647673 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2886505b-3611-4b2c-9760-73cb01d39960-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.647903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.647903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.647903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.648044 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.647986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfqj\" (UniqueName: \"kubernetes.io/projected/2886505b-3611-4b2c-9760-73cb01d39960-kube-api-access-5lfqj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749510 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfqj\" (UniqueName: \"kubernetes.io/projected/2886505b-3611-4b2c-9760-73cb01d39960-kube-api-access-5lfqj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749692 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749753 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2886505b-3611-4b2c-9760-73cb01d39960-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749809 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749869 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749846 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749869 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.749997 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.749876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.750213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.750188 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.750416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.750396 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.752195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.752167 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2886505b-3611-4b2c-9760-73cb01d39960-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.752462 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.752291 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2886505b-3611-4b2c-9760-73cb01d39960-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.757256 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.757233 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfqj\" (UniqueName: \"kubernetes.io/projected/2886505b-3611-4b2c-9760-73cb01d39960-kube-api-access-5lfqj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6v745\" (UID: \"2886505b-3611-4b2c-9760-73cb01d39960\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:08.933040 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:08.932948 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:09.088997 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:09.088900 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6v745"] Jul 09 12:22:09.091726 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:22:09.091691 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2886505b_3611_4b2c_9760_73cb01d39960.slice/crio-ce657a6d011019f164bbae55b020cb9dac796923fbdf79809af924607d3a702d WatchSource:0}: Error finding container ce657a6d011019f164bbae55b020cb9dac796923fbdf79809af924607d3a702d: Status 404 returned error can't find the container with id ce657a6d011019f164bbae55b020cb9dac796923fbdf79809af924607d3a702d Jul 09 12:22:09.355266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:09.355234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" event={"ID":"2886505b-3611-4b2c-9760-73cb01d39960","Type":"ContainerStarted","Data":"ce657a6d011019f164bbae55b020cb9dac796923fbdf79809af924607d3a702d"} Jul 09 12:22:10.362233 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:10.362192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" event={"ID":"2886505b-3611-4b2c-9760-73cb01d39960","Type":"ContainerStarted","Data":"87a2bc011317e33b53c6a568e86918e082ec916eb3b33d8f531d3cea70047ad2"} Jul 09 12:22:10.386753 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:10.386690 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" podStartSLOduration=2.154138516 podStartE2EDuration="2.386671939s" totalImagesPullingTime="232.533423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:22:10.385075788 +0000 UTC m=+1165.147801361" watchObservedRunningTime="2026-07-09 12:22:10.386671939 +0000 UTC m=+1165.149397475" Jul 09 12:22:13.552690 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:13.552657 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:13.553228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:13.552766 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:13.565031 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:13.565007 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:14.394621 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:14.394590 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg" Jul 09 12:22:18.933358 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:18.933315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:18.933834 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:18.933374 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:18.945830 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:18.945805 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:19.420271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:19.420234 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6v745" Jul 09 12:22:46.022841 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.022813 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv"] Jul 09 12:22:46.026706 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.026690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.028835 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.028818 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 12:22:46.035093 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.035071 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv"] Jul 09 12:22:46.198066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.198066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.198265 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.198265 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.198265 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.198372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.198270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dnz\" (UniqueName: \"kubernetes.io/projected/303f2f0a-6deb-4b86-aae3-ad306528f9e4-kube-api-access-k8dnz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.298796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.298796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298739 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.298796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299126 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299126 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299126 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.298869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dnz\" (UniqueName: \"kubernetes.io/projected/303f2f0a-6deb-4b86-aae3-ad306528f9e4-kube-api-access-k8dnz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299292 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.299261 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.299295 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.299437 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.299413 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.301258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.301241 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303f2f0a-6deb-4b86-aae3-ad306528f9e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.301441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.301423 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303f2f0a-6deb-4b86-aae3-ad306528f9e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.308103 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.308082 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dnz\" (UniqueName: \"kubernetes.io/projected/303f2f0a-6deb-4b86-aae3-ad306528f9e4-kube-api-access-k8dnz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv\" (UID: \"303f2f0a-6deb-4b86-aae3-ad306528f9e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.337733 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.337707 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:46.480856 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.480824 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv"] Jul 09 12:22:46.482888 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:22:46.482863 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303f2f0a_6deb_4b86_aae3_ad306528f9e4.slice/crio-3a304d0d6c46035af99992954ff894abda91256f3fd9875261a9862c4786b4cb WatchSource:0}: Error finding container 3a304d0d6c46035af99992954ff894abda91256f3fd9875261a9862c4786b4cb: Status 404 returned error can't find the container with id 3a304d0d6c46035af99992954ff894abda91256f3fd9875261a9862c4786b4cb Jul 09 12:22:46.554019 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:46.553954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" event={"ID":"303f2f0a-6deb-4b86-aae3-ad306528f9e4","Type":"ContainerStarted","Data":"3a304d0d6c46035af99992954ff894abda91256f3fd9875261a9862c4786b4cb"} Jul 09 12:22:47.560989 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:47.560952 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" event={"ID":"303f2f0a-6deb-4b86-aae3-ad306528f9e4","Type":"ContainerStarted","Data":"cefd62dcb9f69b5faae6422d0591ad21a856487d43ecac97d89da142802b3f85"} Jul 09 12:22:47.578027 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:47.577974 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" podStartSLOduration=1.376132342 podStartE2EDuration="1.577957653s" totalImagesPullingTime="201.825311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:22:47.57656292 +0000 UTC m=+1202.339288455" watchObservedRunningTime="2026-07-09 12:22:47.577957653 +0000 UTC m=+1202.340683188" Jul 09 12:22:56.338748 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:56.338712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:56.339141 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:56.338775 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:56.351562 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:56.351534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:22:56.625558 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:22:56.625480 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv" Jul 09 12:23:59.769247 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.769213 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-668576498b-drsxm"] Jul 09 12:23:59.773056 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.773035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.780241 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.780217 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-668576498b-drsxm"] Jul 09 12:23:59.876964 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.876915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a822dd1-5700-4dfa-98d9-45280dce3eb6-tls-cert\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.877152 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.877075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdkd\" (UniqueName: \"kubernetes.io/projected/0a822dd1-5700-4dfa-98d9-45280dce3eb6-kube-api-access-rsdkd\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.978531 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.978498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdkd\" (UniqueName: \"kubernetes.io/projected/0a822dd1-5700-4dfa-98d9-45280dce3eb6-kube-api-access-rsdkd\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.978704 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.978572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a822dd1-5700-4dfa-98d9-45280dce3eb6-tls-cert\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.981079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.981056 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a822dd1-5700-4dfa-98d9-45280dce3eb6-tls-cert\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:23:59.986781 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:23:59.986755 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdkd\" (UniqueName: \"kubernetes.io/projected/0a822dd1-5700-4dfa-98d9-45280dce3eb6-kube-api-access-rsdkd\") pod \"authorino-668576498b-drsxm\" (UID: \"0a822dd1-5700-4dfa-98d9-45280dce3eb6\") " pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:24:00.084144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.084065 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-668576498b-drsxm" Jul 09 12:24:00.217844 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.217818 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-668576498b-drsxm"] Jul 09 12:24:00.219889 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:24:00.219854 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a822dd1_5700_4dfa_98d9_45280dce3eb6.slice/crio-de7a6d94496925a5c9e106b00dd3d4146f6356c4ab8582c63d79bfd6a933dbd2 WatchSource:0}: Error finding container de7a6d94496925a5c9e106b00dd3d4146f6356c4ab8582c63d79bfd6a933dbd2: Status 404 returned error can't find the container with id de7a6d94496925a5c9e106b00dd3d4146f6356c4ab8582c63d79bfd6a933dbd2 Jul 09 12:24:00.221265 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.221240 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:24:00.961161 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.961118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-668576498b-drsxm" event={"ID":"0a822dd1-5700-4dfa-98d9-45280dce3eb6","Type":"ContainerStarted","Data":"3a552f3c1bbdebbe89d1b3a23a7a4b6668fed3f585dc80dd7c4088b9350e6de3"} Jul 09 12:24:00.961622 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.961175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-668576498b-drsxm" event={"ID":"0a822dd1-5700-4dfa-98d9-45280dce3eb6","Type":"ContainerStarted","Data":"de7a6d94496925a5c9e106b00dd3d4146f6356c4ab8582c63d79bfd6a933dbd2"} Jul 09 12:24:00.978967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:00.978899 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-668576498b-drsxm" podStartSLOduration=1.5356092019999998 podStartE2EDuration="1.97888382s" totalImagesPullingTime="443.274618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:24:00.977248262 +0000 UTC m=+1275.739973799" watchObservedRunningTime="2026-07-09 12:24:00.97888382 +0000 UTC m=+1275.741609356" Jul 09 12:24:01.004567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.004534 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:24:01.004756 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.004717 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c7fff-z4kl5" podUID="457671cd-0220-433e-91e4-d88c27a66e9e" containerName="authorino" containerID="cri-o://bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213" gracePeriod=30 Jul 09 12:24:01.273041 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.273006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:24:01.393195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.393166 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert\") pod \"457671cd-0220-433e-91e4-d88c27a66e9e\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " Jul 09 12:24:01.393388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.393226 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkcwp\" (UniqueName: \"kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp\") pod \"457671cd-0220-433e-91e4-d88c27a66e9e\" (UID: \"457671cd-0220-433e-91e4-d88c27a66e9e\") " Jul 09 12:24:01.395399 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.395372 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp" (OuterVolumeSpecName: "kube-api-access-wkcwp") pod "457671cd-0220-433e-91e4-d88c27a66e9e" (UID: "457671cd-0220-433e-91e4-d88c27a66e9e"). InnerVolumeSpecName "kube-api-access-wkcwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:24:01.405354 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.405330 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "457671cd-0220-433e-91e4-d88c27a66e9e" (UID: "457671cd-0220-433e-91e4-d88c27a66e9e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:24:01.494274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.494238 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457671cd-0220-433e-91e4-d88c27a66e9e-tls-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:24:01.494274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.494274 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkcwp\" (UniqueName: \"kubernetes.io/projected/457671cd-0220-433e-91e4-d88c27a66e9e-kube-api-access-wkcwp\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:24:01.968748 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.968710 2567 generic.go:356] "Generic (PLEG): container finished" podID="457671cd-0220-433e-91e4-d88c27a66e9e" containerID="bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213" exitCode=0 Jul 09 12:24:01.968748 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.968751 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c7fff-z4kl5" Jul 09 12:24:01.969314 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.968791 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c7fff-z4kl5" event={"ID":"457671cd-0220-433e-91e4-d88c27a66e9e","Type":"ContainerDied","Data":"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213"} Jul 09 12:24:01.969314 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.968844 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c7fff-z4kl5" event={"ID":"457671cd-0220-433e-91e4-d88c27a66e9e","Type":"ContainerDied","Data":"d7671946414f2b8be99e134da411bd371be12dc1073d72678f23ac0488a29427"} Jul 09 12:24:01.969314 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.968867 2567 scope.go:117] "RemoveContainer" containerID="bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213" Jul 09 12:24:01.979173 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.979152 2567 scope.go:117] "RemoveContainer" containerID="bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213" Jul 09 12:24:01.979490 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:24:01.979463 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213\": container with ID starting with bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213 not found: ID does not exist" containerID="bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213" Jul 09 12:24:01.979566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:01.979505 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213"} err="failed to get container status \"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213\": rpc error: code = NotFound desc = could not find container \"bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213\": container with ID starting with bc3b4760bfabf8dc3b8f99c61d796ab9876361c0db6c93e5118288a098a93213 not found: ID does not exist" Jul 09 12:24:02.020596 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:02.020556 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:24:02.036885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:02.036853 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c7fff-z4kl5"] Jul 09 12:24:03.847885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:24:03.847853 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457671cd-0220-433e-91e4-d88c27a66e9e" path="/var/lib/kubelet/pods/457671cd-0220-433e-91e4-d88c27a66e9e/volumes" Jul 09 12:25:13.199864 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.199769 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:25:13.200491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.200004 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" podUID="74b703b6-6544-4edb-9d3f-33738d7f5053" containerName="manager" containerID="cri-o://0d274c7e220fdbd31d1a48d164e413af1220ee9715ed67f4b4c045641c676472" gracePeriod=30 Jul 09 12:25:13.349238 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.349197 2567 generic.go:356] "Generic (PLEG): container finished" podID="74b703b6-6544-4edb-9d3f-33738d7f5053" containerID="0d274c7e220fdbd31d1a48d164e413af1220ee9715ed67f4b4c045641c676472" exitCode=0 Jul 09 12:25:13.349394 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.349267 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" event={"ID":"74b703b6-6544-4edb-9d3f-33738d7f5053","Type":"ContainerDied","Data":"0d274c7e220fdbd31d1a48d164e413af1220ee9715ed67f4b4c045641c676472"} Jul 09 12:25:13.453315 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.453260 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:25:13.558755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.558725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") pod \"74b703b6-6544-4edb-9d3f-33738d7f5053\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " Jul 09 12:25:13.558941 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.558802 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfbh7\" (UniqueName: \"kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7\") pod \"74b703b6-6544-4edb-9d3f-33738d7f5053\" (UID: \"74b703b6-6544-4edb-9d3f-33738d7f5053\") " Jul 09 12:25:13.560979 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.560953 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7" (OuterVolumeSpecName: "kube-api-access-wfbh7") pod "74b703b6-6544-4edb-9d3f-33738d7f5053" (UID: "74b703b6-6544-4edb-9d3f-33738d7f5053"). InnerVolumeSpecName "kube-api-access-wfbh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:25:13.561096 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.560954 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "74b703b6-6544-4edb-9d3f-33738d7f5053" (UID: "74b703b6-6544-4edb-9d3f-33738d7f5053"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:25:13.660130 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.660095 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbh7\" (UniqueName: \"kubernetes.io/projected/74b703b6-6544-4edb-9d3f-33738d7f5053-kube-api-access-wfbh7\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:25:13.660130 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:13.660124 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74b703b6-6544-4edb-9d3f-33738d7f5053-webhook-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:25:14.356313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:14.356277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" event={"ID":"74b703b6-6544-4edb-9d3f-33738d7f5053","Type":"ContainerDied","Data":"5ccfabddef6b05ceccc3c6cab9136ec8302da56299aacfed9b7045baecc9b90e"} Jul 09 12:25:14.356313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:14.356302 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-mpvb7" Jul 09 12:25:14.356802 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:14.356341 2567 scope.go:117] "RemoveContainer" containerID="0d274c7e220fdbd31d1a48d164e413af1220ee9715ed67f4b4c045641c676472" Jul 09 12:25:14.374421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:14.374394 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:25:14.378806 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:14.378780 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-mpvb7"] Jul 09 12:25:15.130108 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130075 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f8c776fc-lsz4x"] Jul 09 12:25:15.130524 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130509 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b703b6-6544-4edb-9d3f-33738d7f5053" containerName="manager" Jul 09 12:25:15.130599 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130529 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b703b6-6544-4edb-9d3f-33738d7f5053" containerName="manager" Jul 09 12:25:15.130599 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130542 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457671cd-0220-433e-91e4-d88c27a66e9e" containerName="authorino" Jul 09 12:25:15.130599 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130548 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="457671cd-0220-433e-91e4-d88c27a66e9e" containerName="authorino" Jul 09 12:25:15.130719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130651 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b703b6-6544-4edb-9d3f-33738d7f5053" containerName="manager" Jul 09 12:25:15.130719 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.130674 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="457671cd-0220-433e-91e4-d88c27a66e9e" containerName="authorino" Jul 09 12:25:15.134388 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.134370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.136983 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.136966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 12:25:15.137860 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.137845 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dwnjj\"" Jul 09 12:25:15.137974 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.137939 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 12:25:15.142821 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.142797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-lsz4x"] Jul 09 12:25:15.275900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.275859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-webhook-cert\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.276076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.275900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsj8s\" (UniqueName: \"kubernetes.io/projected/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-kube-api-access-gsj8s\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.377259 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.377229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-webhook-cert\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.377259 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.377262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsj8s\" (UniqueName: \"kubernetes.io/projected/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-kube-api-access-gsj8s\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.379903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.379878 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-webhook-cert\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.386811 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.386747 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsj8s\" (UniqueName: \"kubernetes.io/projected/3c0a5098-1a51-4c25-9cb2-bb9c020bba03-kube-api-access-gsj8s\") pod \"maas-controller-79f8c776fc-lsz4x\" (UID: \"3c0a5098-1a51-4c25-9cb2-bb9c020bba03\") " pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.445990 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.445959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:15.584335 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.584307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8c776fc-lsz4x"] Jul 09 12:25:15.586550 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:25:15.586524 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0a5098_1a51_4c25_9cb2_bb9c020bba03.slice/crio-d84697f1da626fbca34a66fc57c4e17e0dbbcb83632e8afe6acdad2a6dd8c59f WatchSource:0}: Error finding container d84697f1da626fbca34a66fc57c4e17e0dbbcb83632e8afe6acdad2a6dd8c59f: Status 404 returned error can't find the container with id d84697f1da626fbca34a66fc57c4e17e0dbbcb83632e8afe6acdad2a6dd8c59f Jul 09 12:25:15.848760 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:15.848729 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b703b6-6544-4edb-9d3f-33738d7f5053" path="/var/lib/kubelet/pods/74b703b6-6544-4edb-9d3f-33738d7f5053/volumes" Jul 09 12:25:16.370228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:16.370141 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" event={"ID":"3c0a5098-1a51-4c25-9cb2-bb9c020bba03","Type":"ContainerStarted","Data":"8ee8ed540a965262e9698eaf3132b8e27f17d273464bf1a164df4cdde136acf0"} Jul 09 12:25:16.370228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:16.370190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" event={"ID":"3c0a5098-1a51-4c25-9cb2-bb9c020bba03","Type":"ContainerStarted","Data":"d84697f1da626fbca34a66fc57c4e17e0dbbcb83632e8afe6acdad2a6dd8c59f"} Jul 09 12:25:16.370461 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:16.370315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:25:16.388870 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:16.388811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" podStartSLOduration=0.892396033 podStartE2EDuration="1.388791522s" totalImagesPullingTime="496.395489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:25:16.388574709 +0000 UTC m=+1351.151300266" watchObservedRunningTime="2026-07-09 12:25:16.388791522 +0000 UTC m=+1351.151517058" Jul 09 12:25:27.386176 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:25:27.386147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f8c776fc-lsz4x" Jul 09 12:26:45.838102 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:26:45.838073 2567 scope.go:117] "RemoveContainer" containerID="52f41c19ab318444d2a6444d0d7991419f24bd16e70d2d2cf1ec64295792360d" Jul 09 12:30:00.148409 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.148373 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:30:00.151751 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.151736 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:30:00.154386 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.154368 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-52h8m\"" Jul 09 12:30:00.168324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.168302 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:30:00.276650 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.276611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbf8t\" (UniqueName: \"kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t\") pod \"maas-api-key-cleanup-29726670-kzhc4\" (UID: \"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:30:00.377128 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.377099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbf8t\" (UniqueName: \"kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t\") pod \"maas-api-key-cleanup-29726670-kzhc4\" (UID: \"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:30:00.385855 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.385825 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbf8t\" (UniqueName: \"kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t\") pod \"maas-api-key-cleanup-29726670-kzhc4\" (UID: \"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:30:00.462002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.461968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:30:00.599729 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.599705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:30:00.601476 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:30:00.601449 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2332405_ef7d_49c4_b5fb_4e2faaaeb76f.slice/crio-c24c2b2ad8b943dfd406072efdebca7f3006146f168d80f4796918c027f7db26 WatchSource:0}: Error finding container c24c2b2ad8b943dfd406072efdebca7f3006146f168d80f4796918c027f7db26: Status 404 returned error can't find the container with id c24c2b2ad8b943dfd406072efdebca7f3006146f168d80f4796918c027f7db26 Jul 09 12:30:00.603386 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.603369 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:30:00.879017 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:00.878912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" event={"ID":"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f","Type":"ContainerStarted","Data":"c24c2b2ad8b943dfd406072efdebca7f3006146f168d80f4796918c027f7db26"} Jul 09 12:30:01.384049 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:01.384010 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:30:01.384544 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:01.384227 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" podUID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" containerName="manager" containerID="cri-o://6c089a5b540b4ef0e5417b1acd0ff2309c9a1c136ba996e4db19ea563aad9335" gracePeriod=10 Jul 09 12:30:05.911131 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:05.911086 2567 generic.go:356] "Generic (PLEG): container finished" podID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" containerID="6c089a5b540b4ef0e5417b1acd0ff2309c9a1c136ba996e4db19ea563aad9335" exitCode=0 Jul 09 12:30:05.911529 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:05.911192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" event={"ID":"b1f35421-8d4e-4048-8c90-d0cfab2285bd","Type":"ContainerDied","Data":"6c089a5b540b4ef0e5417b1acd0ff2309c9a1c136ba996e4db19ea563aad9335"} Jul 09 12:30:05.939619 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:05.939599 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:30:06.033236 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.033204 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hjjh\" (UniqueName: \"kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh\") pod \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " Jul 09 12:30:06.033427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.033310 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume\") pod \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\" (UID: \"b1f35421-8d4e-4048-8c90-d0cfab2285bd\") " Jul 09 12:30:06.033685 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.033661 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1f35421-8d4e-4048-8c90-d0cfab2285bd" (UID: "b1f35421-8d4e-4048-8c90-d0cfab2285bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:30:06.035503 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.035482 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh" (OuterVolumeSpecName: "kube-api-access-7hjjh") pod "b1f35421-8d4e-4048-8c90-d0cfab2285bd" (UID: "b1f35421-8d4e-4048-8c90-d0cfab2285bd"). InnerVolumeSpecName "kube-api-access-7hjjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:30:06.134121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.134088 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f35421-8d4e-4048-8c90-d0cfab2285bd-extensions-socket-volume\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:30:06.134121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.134113 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjjh\" (UniqueName: \"kubernetes.io/projected/b1f35421-8d4e-4048-8c90-d0cfab2285bd-kube-api-access-7hjjh\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:30:06.918129 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.918078 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" event={"ID":"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f","Type":"ContainerStarted","Data":"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6"} Jul 09 12:30:06.919465 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.919445 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" Jul 09 12:30:06.919584 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.919449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2" event={"ID":"b1f35421-8d4e-4048-8c90-d0cfab2285bd","Type":"ContainerDied","Data":"d70ae585a0419b75bec79f65b184b27d0c600517cab0e79e2724f8b531a453df"} Jul 09 12:30:06.919584 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.919576 2567 scope.go:117] "RemoveContainer" containerID="6c089a5b540b4ef0e5417b1acd0ff2309c9a1c136ba996e4db19ea563aad9335" Jul 09 12:30:06.933564 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.933522 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" podStartSLOduration=1.691597563 podStartE2EDuration="6.933505094s" totalImagesPullingTime="5.241907531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:30:06.93317125 +0000 UTC m=+1641.695896787" watchObservedRunningTime="2026-07-09 12:30:06.933505094 +0000 UTC m=+1641.696230631" Jul 09 12:30:06.946774 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.946751 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:30:06.950627 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:06.950607 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mzkp2"] Jul 09 12:30:07.847904 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:30:07.847869 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" path="/var/lib/kubelet/pods/b1f35421-8d4e-4048-8c90-d0cfab2285bd/volumes" Jul 09 12:31:00.011637 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:00.011596 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:31:00.012288 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:00.011796 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" podUID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" containerName="cleanup" containerID="cri-o://cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6" gracePeriod=30 Jul 09 12:31:02.515514 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.515479 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd"] Jul 09 12:31:02.516120 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.516100 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" containerName="manager" Jul 09 12:31:02.516120 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.516121 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" containerName="manager" Jul 09 12:31:02.516262 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.516250 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f35421-8d4e-4048-8c90-d0cfab2285bd" containerName="manager" Jul 09 12:31:02.519325 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.519303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.522337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.522317 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qstkz\"" Jul 09 12:31:02.530312 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.530287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd"] Jul 09 12:31:02.546563 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.546534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0154982d-1ece-4dea-82a3-2f3bbe262038-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.546687 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.546649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fc7d\" (UniqueName: \"kubernetes.io/projected/0154982d-1ece-4dea-82a3-2f3bbe262038-kube-api-access-7fc7d\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.647609 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.647560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fc7d\" (UniqueName: \"kubernetes.io/projected/0154982d-1ece-4dea-82a3-2f3bbe262038-kube-api-access-7fc7d\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.647796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.647629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0154982d-1ece-4dea-82a3-2f3bbe262038-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.648020 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.648005 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0154982d-1ece-4dea-82a3-2f3bbe262038-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.655555 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.655536 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fc7d\" (UniqueName: \"kubernetes.io/projected/0154982d-1ece-4dea-82a3-2f3bbe262038-kube-api-access-7fc7d\") pod \"kuadrant-operator-controller-manager-86878c769f-ksbgd\" (UID: \"0154982d-1ece-4dea-82a3-2f3bbe262038\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.830214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.830119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:02.978706 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:02.978679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd"] Jul 09 12:31:02.980497 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:31:02.980466 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0154982d_1ece_4dea_82a3_2f3bbe262038.slice/crio-d9d13f8e3e65fd5e9a789bf7e06c56e41c51650c8e6ed0b2292da2027f451b16 WatchSource:0}: Error finding container d9d13f8e3e65fd5e9a789bf7e06c56e41c51650c8e6ed0b2292da2027f451b16: Status 404 returned error can't find the container with id d9d13f8e3e65fd5e9a789bf7e06c56e41c51650c8e6ed0b2292da2027f451b16 Jul 09 12:31:03.205884 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:03.205841 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" event={"ID":"0154982d-1ece-4dea-82a3-2f3bbe262038","Type":"ContainerStarted","Data":"4e7151c9f8b5ea5c7ff18b8b485f8656f2ad16fb076d883fb6a0d1425e87c9ec"} Jul 09 12:31:03.206132 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:03.205923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" event={"ID":"0154982d-1ece-4dea-82a3-2f3bbe262038","Type":"ContainerStarted","Data":"d9d13f8e3e65fd5e9a789bf7e06c56e41c51650c8e6ed0b2292da2027f451b16"} Jul 09 12:31:03.206132 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:03.205982 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:03.223701 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:03.223644 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" podStartSLOduration=1.223628333 podStartE2EDuration="1.223628333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:31:03.222890535 +0000 UTC m=+1697.985616082" watchObservedRunningTime="2026-07-09 12:31:03.223628333 +0000 UTC m=+1697.986353867" Jul 09 12:31:14.213731 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:14.213703 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ksbgd" Jul 09 12:31:30.202560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.202536 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:31:30.306138 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.306046 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbf8t\" (UniqueName: \"kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t\") pod \"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f\" (UID: \"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f\") " Jul 09 12:31:30.308375 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.308341 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t" (OuterVolumeSpecName: "kube-api-access-hbf8t") pod "b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" (UID: "b2332405-ef7d-49c4-b5fb-4e2faaaeb76f"). InnerVolumeSpecName "kube-api-access-hbf8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:31:30.356616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.356581 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" containerID="cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6" exitCode=137 Jul 09 12:31:30.356763 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.356629 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" Jul 09 12:31:30.356763 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.356666 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" event={"ID":"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f","Type":"ContainerDied","Data":"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6"} Jul 09 12:31:30.356763 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.356727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4" event={"ID":"b2332405-ef7d-49c4-b5fb-4e2faaaeb76f","Type":"ContainerDied","Data":"c24c2b2ad8b943dfd406072efdebca7f3006146f168d80f4796918c027f7db26"} Jul 09 12:31:30.356763 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.356750 2567 scope.go:117] "RemoveContainer" containerID="cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6" Jul 09 12:31:30.366918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.366902 2567 scope.go:117] "RemoveContainer" containerID="cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6" Jul 09 12:31:30.367279 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:31:30.367249 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6\": container with ID starting with cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6 not found: ID does not exist" containerID="cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6" Jul 09 12:31:30.367355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.367291 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6"} err="failed to get container status \"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6\": rpc error: code = NotFound desc = could not find container \"cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6\": container with ID starting with cd86478e7ebd313aaa4ff22273b9d2ab25781f588603e919c239ed38dc5ee3e6 not found: ID does not exist" Jul 09 12:31:30.381872 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.381838 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:31:30.387570 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.387546 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726670-kzhc4"] Jul 09 12:31:30.407533 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:30.407500 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbf8t\" (UniqueName: \"kubernetes.io/projected/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f-kube-api-access-hbf8t\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:31:31.848922 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:31:31.848890 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" path="/var/lib/kubelet/pods/b2332405-ef7d-49c4-b5fb-4e2faaaeb76f/volumes" Jul 09 12:37:33.072197 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.072163 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:33.072836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.072818 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" containerName="cleanup" Jul 09 12:37:33.072886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.072840 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" containerName="cleanup" Jul 09 12:37:33.072994 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.072969 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2332405-ef7d-49c4-b5fb-4e2faaaeb76f" containerName="cleanup" Jul 09 12:37:33.076697 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.076653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.079369 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.079346 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 12:37:33.079625 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.079574 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f774bd66-openshift-default-dockercfg-lsf28\"" Jul 09 12:37:33.086430 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.086410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:33.227395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwph\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227503 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227566 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227521 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227826 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227615 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227826 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227676 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.227826 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.227727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333541 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333715 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333715 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333672 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333715 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333898 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333898 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.333898 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwph\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334247 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.334214 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334306 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.334275 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334391 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.333966 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334505 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.334440 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.334640 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.334558 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.336533 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.336509 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.336807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.336788 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.347658 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.347636 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwph\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.349416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.349394 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token\") pod \"e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.389547 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.389516 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:33.544622 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.544585 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:33.546301 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:37:33.546274 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cbbd03_af15_4c8b_a91d_b92c405f80e3.slice/crio-ffb3af42b87d54c77843bfa451cbb054cb5290d5dfd6ce62477b45915a483582 WatchSource:0}: Error finding container ffb3af42b87d54c77843bfa451cbb054cb5290d5dfd6ce62477b45915a483582: Status 404 returned error can't find the container with id ffb3af42b87d54c77843bfa451cbb054cb5290d5dfd6ce62477b45915a483582 Jul 09 12:37:33.548690 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:33.548673 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:37:34.277005 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:34.276968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerStarted","Data":"ffb3af42b87d54c77843bfa451cbb054cb5290d5dfd6ce62477b45915a483582"} Jul 09 12:37:36.289830 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:36.289792 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:36.290147 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:36.289862 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:36.290147 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:36.289900 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:37.297618 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:37.297580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerStarted","Data":"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105"} Jul 09 12:37:37.315804 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:37.315752 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" podStartSLOduration=1.575086825 podStartE2EDuration="4.31573818s" totalImagesPullingTime="2.740651355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:37:37.31487612 +0000 UTC m=+2092.077601656" watchObservedRunningTime="2026-07-09 12:37:37.31573818 +0000 UTC m=+2092.078463715" Jul 09 12:37:37.390648 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:37.390617 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:37.394922 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:37.394896 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:38.303075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:38.303042 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:38.369314 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:38.369288 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:45.342600 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.342571 2567 generic.go:356] "Generic (PLEG): container finished" podID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerID="38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105" exitCode=0 Jul 09 12:37:45.343048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.342644 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerDied","Data":"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105"} Jul 09 12:37:45.343048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.343025 2567 scope.go:117] "RemoveContainer" containerID="38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105" Jul 09 12:37:45.345097 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.345060 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:45.345187 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.345133 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:45.345187 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.345165 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:37:45.530480 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:45.530427 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:46.350900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:46.350867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerStarted","Data":"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934"} Jul 09 12:37:46.351324 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:46.350956 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" containerID="cri-o://505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934" gracePeriod=30 Jul 09 12:37:51.600230 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.600203 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:51.617487 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617426 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617487 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617462 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617488 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617518 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617551 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617601 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617660 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617690 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtwph\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.617865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617695 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:37:51.617865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617727 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket\") pod \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\" (UID: \"e1cbbd03-af15-4c8b-a91d-b92c405f80e3\") " Jul 09 12:37:51.618125 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.617979 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:37:51.618675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.618179 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.618675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.618200 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.618675 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.618259 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:37:51.619296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.619266 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:37:51.620055 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.619913 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:37:51.620880 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.620438 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:37:51.620880 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.620506 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:37:51.621072 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.620887 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:37:51.621072 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.620886 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph" (OuterVolumeSpecName: "kube-api-access-rtwph") pod "e1cbbd03-af15-4c8b-a91d-b92c405f80e3" (UID: "e1cbbd03-af15-4c8b-a91d-b92c405f80e3"). InnerVolumeSpecName "kube-api-access-rtwph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:37:51.719378 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719350 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719378 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719373 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719378 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719381 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719389 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719396 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719406 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwph\" (UniqueName: \"kubernetes.io/projected/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-kube-api-access-rtwph\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:51.719610 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:51.719417 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1cbbd03-af15-4c8b-a91d-b92c405f80e3-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:37:52.385587 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.385555 2567 generic.go:356] "Generic (PLEG): container finished" podID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerID="505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934" exitCode=0 Jul 09 12:37:52.385758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.385611 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" Jul 09 12:37:52.385758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.385632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerDied","Data":"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934"} Jul 09 12:37:52.385758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.385687 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw" event={"ID":"e1cbbd03-af15-4c8b-a91d-b92c405f80e3","Type":"ContainerDied","Data":"ffb3af42b87d54c77843bfa451cbb054cb5290d5dfd6ce62477b45915a483582"} Jul 09 12:37:52.385758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.385711 2567 scope.go:117] "RemoveContainer" containerID="505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934" Jul 09 12:37:52.395502 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.395485 2567 scope.go:117] "RemoveContainer" containerID="38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105" Jul 09 12:37:52.404351 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.404335 2567 scope.go:117] "RemoveContainer" containerID="505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934" Jul 09 12:37:52.404626 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:37:52.404607 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934\": container with ID starting with 505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934 not found: ID does not exist" containerID="505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934" Jul 09 12:37:52.404766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.404638 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934"} err="failed to get container status \"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934\": rpc error: code = NotFound desc = could not find container \"505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934\": container with ID starting with 505c651860668ff101733241364bb5c1ffc8deed907b72d5c1313ba978d62934 not found: ID does not exist" Jul 09 12:37:52.404766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.404661 2567 scope.go:117] "RemoveContainer" containerID="38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105" Jul 09 12:37:52.404903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.404878 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:52.405017 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:37:52.404983 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105\": container with ID starting with 38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105 not found: ID does not exist" containerID="38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105" Jul 09 12:37:52.405017 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.405022 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105"} err="failed to get container status \"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105\": rpc error: code = NotFound desc = could not find container \"38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105\": container with ID starting with 38639536dc5f192d8bf3271964d0a01e48ab98bcbc36f4c3403c0004114ca105 not found: ID does not exist" Jul 09 12:37:52.407558 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:52.407527 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f774bd66-openshift-default-7c48f776c-cbpcw"] Jul 09 12:37:53.847896 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:37:53.847867 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" path="/var/lib/kubelet/pods/e1cbbd03-af15-4c8b-a91d-b92c405f80e3/volumes" Jul 09 12:39:40.441578 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.441550 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:39:40.442075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.442061 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:39:40.442126 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.442078 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:39:40.442172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.442163 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:39:40.442208 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.442176 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:39:40.445489 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.445474 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.448128 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.448107 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 12:39:40.448273 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.448164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2cff3f7c-openshift-default-dockercfg-w4qrk\"" Jul 09 12:39:40.462793 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.461899 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:39:40.622196 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622410 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fdm\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622410 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622410 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622410 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622370 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622591 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622591 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622591 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.622703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.622589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724411 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724411 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724411 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fdm\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.724561 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.725232 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.725193 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.725481 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.725457 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.729435 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.729387 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.732851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.729620 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.732851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.724619 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.732851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.729892 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.733121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.733095 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.736256 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.736227 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.736471 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.736451 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fdm\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm\") pod \"e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.756400 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.756382 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:40.905441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.905417 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:39:40.907183 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:39:40.907155 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c9ec88_1974_4f17_8569_740ec760de19.slice/crio-8b160cfe2fda09cd9cb588d822b6abfb9b31da55e800861ec9173e6777e6a630 WatchSource:0}: Error finding container 8b160cfe2fda09cd9cb588d822b6abfb9b31da55e800861ec9173e6777e6a630: Status 404 returned error can't find the container with id 8b160cfe2fda09cd9cb588d822b6abfb9b31da55e800861ec9173e6777e6a630 Jul 09 12:39:40.909255 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.909225 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:40.909341 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.909286 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:40.909341 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.909313 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:40.940216 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:40.940188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerStarted","Data":"8b160cfe2fda09cd9cb588d822b6abfb9b31da55e800861ec9173e6777e6a630"} Jul 09 12:39:41.946698 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:41.946663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerStarted","Data":"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49"} Jul 09 12:39:41.970339 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:41.970281 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podStartSLOduration=1.9702609340000001 podStartE2EDuration="1.970260934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:39:41.969198952 +0000 UTC m=+2216.731924515" watchObservedRunningTime="2026-07-09 12:39:41.970260934 +0000 UTC m=+2216.732986470" Jul 09 12:39:42.756868 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:42.756827 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:42.761656 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:42.761630 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:42.952025 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:42.951998 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:42.952917 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:42.952897 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:57.953811 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:57.953778 2567 patch_prober.go:28] interesting pod/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 12:39:57.954266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:57.953832 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 12:39:58.045455 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.045424 2567 generic.go:356] "Generic (PLEG): container finished" podID="c3c9ec88-1974-4f17-8569-740ec760de19" containerID="842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49" exitCode=0 Jul 09 12:39:58.045652 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.045499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerDied","Data":"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49"} Jul 09 12:39:58.045967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.045944 2567 scope.go:117] "RemoveContainer" containerID="842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49" Jul 09 12:39:58.047087 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.047050 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:58.047228 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.047206 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:58.047327 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:58.047308 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:39:59.053631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:59.053587 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerStarted","Data":"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b"} Jul 09 12:39:59.757504 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:59.757469 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:39:59.757680 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:59.757649 2567 patch_prober.go:28] interesting pod/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 12:39:59.757726 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:39:59.757706 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 12:40:00.757578 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:00.757541 2567 patch_prober.go:28] interesting pod/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 12:40:00.758085 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:00.757600 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 12:40:01.757347 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:01.757307 2567 patch_prober.go:28] interesting pod/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 12:40:01.757608 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:01.757370 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 12:40:02.429753 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:02.429669 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:40:02.430010 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:02.429983 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" containerID="cri-o://da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b" gracePeriod=30 Jul 09 12:40:08.399155 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.399122 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:40:08.509310 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509278 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509343 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509370 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509397 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509505 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509529 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509556 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509792 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8fdm\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.509792 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509716 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:40:08.509906 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.509828 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:40:08.510559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510050 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:40:08.510559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510309 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket\") pod \"c3c9ec88-1974-4f17-8569-740ec760de19\" (UID: \"c3c9ec88-1974-4f17-8569-740ec760de19\") " Jul 09 12:40:08.510559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510506 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:40:08.510559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510520 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:40:08.510822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510727 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.510822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510740 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.510822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510750 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.510822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510758 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.510822 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.510765 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c9ec88-1974-4f17-8569-740ec760de19-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.511807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.511777 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:40:08.511988 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.511966 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:40:08.512266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.512246 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:40:08.512365 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.512342 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm" (OuterVolumeSpecName: "kube-api-access-n8fdm") pod "c3c9ec88-1974-4f17-8569-740ec760de19" (UID: "c3c9ec88-1974-4f17-8569-740ec760de19"). InnerVolumeSpecName "kube-api-access-n8fdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:40:08.611322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.611289 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.611322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.611315 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c9ec88-1974-4f17-8569-740ec760de19-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.611322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.611324 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c9ec88-1974-4f17-8569-740ec760de19-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:08.611546 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:08.611332 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fdm\" (UniqueName: \"kubernetes.io/projected/c3c9ec88-1974-4f17-8569-740ec760de19-kube-api-access-n8fdm\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:40:09.115381 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.115341 2567 generic.go:356] "Generic (PLEG): container finished" podID="c3c9ec88-1974-4f17-8569-740ec760de19" containerID="da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b" exitCode=0 Jul 09 12:40:09.115549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.115388 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerDied","Data":"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b"} Jul 09 12:40:09.115549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.115397 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" Jul 09 12:40:09.115549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.115446 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f" event={"ID":"c3c9ec88-1974-4f17-8569-740ec760de19","Type":"ContainerDied","Data":"8b160cfe2fda09cd9cb588d822b6abfb9b31da55e800861ec9173e6777e6a630"} Jul 09 12:40:09.115549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.115478 2567 scope.go:117] "RemoveContainer" containerID="da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b" Jul 09 12:40:09.124849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.124831 2567 scope.go:117] "RemoveContainer" containerID="842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49" Jul 09 12:40:09.133002 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.132983 2567 scope.go:117] "RemoveContainer" containerID="da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b" Jul 09 12:40:09.133249 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:40:09.133222 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b\": container with ID starting with da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b not found: ID does not exist" containerID="da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b" Jul 09 12:40:09.133300 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.133260 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b"} err="failed to get container status \"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b\": rpc error: code = NotFound desc = could not find container \"da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b\": container with ID starting with da29e0395c6ec12ed1068c443741c32c73a7172186da9b86e828346e9ba86b1b not found: ID does not exist" Jul 09 12:40:09.133300 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.133280 2567 scope.go:117] "RemoveContainer" containerID="842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49" Jul 09 12:40:09.133529 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:40:09.133513 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49\": container with ID starting with 842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49 not found: ID does not exist" containerID="842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49" Jul 09 12:40:09.133580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.133533 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49"} err="failed to get container status \"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49\": rpc error: code = NotFound desc = could not find container \"842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49\": container with ID starting with 842ad92ac2a5f9382319d5a09b31d1f379ca89e6ebf20d0588aac09430d59d49 not found: ID does not exist" Jul 09 12:40:09.136550 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.136530 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:40:09.142530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.142510 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2cff3f7c-openshift-default-5b68b4759f-6bg8f"] Jul 09 12:40:09.848611 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:09.848574 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" path="/var/lib/kubelet/pods/c3c9ec88-1974-4f17-8569-740ec760de19/volumes" Jul 09 12:40:22.232915 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.232882 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:40:22.233579 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233564 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.233635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233582 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.233635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233601 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.233635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233609 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.233635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233623 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:40:22.233635 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233631 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cbbd03-af15-4c8b-a91d-b92c405f80e3" containerName="istio-proxy" Jul 09 12:40:22.233836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233729 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.233836 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.233742 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c9ec88-1974-4f17-8569-740ec760de19" containerName="istio-proxy" Jul 09 12:40:22.237229 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.237212 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.239328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.239307 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 12:40:22.239442 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.239351 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc67cdd8-openshift-default-dockercfg-m47j6\"" Jul 09 12:40:22.246842 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.246817 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:40:22.343858 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.343825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.343892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344049 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344081 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344252 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55xt\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344252 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344252 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.344402 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.344306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445318 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445440 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55xt\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445690 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445690 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445690 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445833 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.445966 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445898 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.446037 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.445997 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.446092 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.446048 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.446148 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.446136 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.448145 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.448118 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.448290 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.448275 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.452944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.452914 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.453207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.453190 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55xt\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt\") pod \"e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.548275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.548194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:22.698062 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.697855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:40:22.700553 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:40:22.700519 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054a0ab6_9818_4f2c_97ad_d82a1b5493ac.slice/crio-ed180cdad1b90029b1c12c32326cb456f57c51dfe6ee7704184481c3fb73368c WatchSource:0}: Error finding container ed180cdad1b90029b1c12c32326cb456f57c51dfe6ee7704184481c3fb73368c: Status 404 returned error can't find the container with id ed180cdad1b90029b1c12c32326cb456f57c51dfe6ee7704184481c3fb73368c Jul 09 12:40:22.702983 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.702921 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:22.703095 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.703016 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:22.703095 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:22.703044 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:23.189687 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:23.189657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerStarted","Data":"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a"} Jul 09 12:40:23.189851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:23.189706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerStarted","Data":"ed180cdad1b90029b1c12c32326cb456f57c51dfe6ee7704184481c3fb73368c"} Jul 09 12:40:23.208773 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:23.208730 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podStartSLOduration=1.208715022 podStartE2EDuration="1.208715022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:40:23.207459651 +0000 UTC m=+2257.970185189" watchObservedRunningTime="2026-07-09 12:40:23.208715022 +0000 UTC m=+2257.971440557" Jul 09 12:40:23.548678 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:23.548597 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:23.553227 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:23.553205 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:24.195547 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:24.195505 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:24.196626 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:24.196605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:39.196877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.196842 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 09 12:40:39.197321 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.196910 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 09 12:40:39.276510 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.276478 2567 generic.go:356] "Generic (PLEG): container finished" podID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerID="b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a" exitCode=0 Jul 09 12:40:39.276678 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.276515 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerDied","Data":"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a"} Jul 09 12:40:39.276992 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.276978 2567 scope.go:117] "RemoveContainer" containerID="b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a" Jul 09 12:40:39.278197 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.278168 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:39.278271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.278254 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:39.278313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:39.278299 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:40:40.287056 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:40.287017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerStarted","Data":"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e"} Jul 09 12:40:40.549225 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:40.549129 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:40.549379 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:40.549346 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 09 12:40:40.549423 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:40.549392 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 09 12:40:41.548847 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:41.548811 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 09 12:40:41.549267 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:41.548861 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 09 12:40:42.549500 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:42.549465 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 09 12:40:42.549923 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:42.549518 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 09 12:40:43.548758 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:43.548711 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 09 12:40:43.548981 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:43.548781 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 09 12:40:44.553333 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:44.553301 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:44.553753 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:44.553691 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:40:44.554504 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:40:44.554486 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:42:12.931471 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:12.931435 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:42:12.932001 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:12.931679 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" containerID="cri-o://8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e" gracePeriod=30 Jul 09 12:42:14.555864 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:14.555828 2567 patch_prober.go:28] interesting pod/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 12:42:14.556261 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:14.555895 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 12:42:18.198798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.198733 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:42:18.267461 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267423 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267461 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267463 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267487 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267523 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267558 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267604 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55xt\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267629 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267666 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.267723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267707 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data\") pod \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\" (UID: \"054a0ab6-9818-4f2c-97ad-d82a1b5493ac\") " Jul 09 12:42:18.268140 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.267885 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:42:18.268140 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.268106 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.268370 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.268347 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:42:18.268494 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.268476 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:42:18.268618 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.268595 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:42:18.268799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.268778 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:42:18.270613 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.270579 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:42:18.270613 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.270595 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt" (OuterVolumeSpecName: "kube-api-access-w55xt") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "kube-api-access-w55xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:42:18.270855 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.270726 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:42:18.271104 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.271069 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "054a0ab6-9818-4f2c-97ad-d82a1b5493ac" (UID: "054a0ab6-9818-4f2c-97ad-d82a1b5493ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:42:18.368810 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368778 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.368810 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368804 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.368810 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368813 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.368810 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368822 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.368810 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368833 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55xt\" (UniqueName: \"kubernetes.io/projected/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-kube-api-access-w55xt\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.369233 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368843 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.369233 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368851 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.369233 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.368859 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a0ab6-9818-4f2c-97ad-d82a1b5493ac-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:42:18.821855 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.821820 2567 generic.go:356] "Generic (PLEG): container finished" podID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerID="8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e" exitCode=0 Jul 09 12:42:18.822058 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.821892 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" Jul 09 12:42:18.822058 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.821903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerDied","Data":"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e"} Jul 09 12:42:18.822058 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.821972 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm" event={"ID":"054a0ab6-9818-4f2c-97ad-d82a1b5493ac","Type":"ContainerDied","Data":"ed180cdad1b90029b1c12c32326cb456f57c51dfe6ee7704184481c3fb73368c"} Jul 09 12:42:18.822058 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.821995 2567 scope.go:117] "RemoveContainer" containerID="8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e" Jul 09 12:42:18.832106 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.832091 2567 scope.go:117] "RemoveContainer" containerID="b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a" Jul 09 12:42:18.841342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.841313 2567 scope.go:117] "RemoveContainer" containerID="8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e" Jul 09 12:42:18.841640 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:42:18.841617 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e\": container with ID starting with 8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e not found: ID does not exist" containerID="8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e" Jul 09 12:42:18.841730 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.841648 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e"} err="failed to get container status \"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e\": rpc error: code = NotFound desc = could not find container \"8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e\": container with ID starting with 8c7fe7bd552e6945d00e95179878c592b6b973f6da32b439de862f40b8367f4e not found: ID does not exist" Jul 09 12:42:18.841730 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.841671 2567 scope.go:117] "RemoveContainer" containerID="b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a" Jul 09 12:42:18.842025 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:42:18.842003 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a\": container with ID starting with b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a not found: ID does not exist" containerID="b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a" Jul 09 12:42:18.842113 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.842030 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a"} err="failed to get container status \"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a\": rpc error: code = NotFound desc = could not find container \"b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a\": container with ID starting with b2c808e9dd555d2721f169a34e902283898a5ef2787105ef600a4d15d1f29b3a not found: ID does not exist" Jul 09 12:42:18.847982 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.847945 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:42:18.851189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:18.851165 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc67cdd8-openshift-default-575d59f87-mmfdm"] Jul 09 12:42:19.850067 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:42:19.850036 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" path="/var/lib/kubelet/pods/054a0ab6-9818-4f2c-97ad-d82a1b5493ac/volumes" Jul 09 12:43:08.695801 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.695723 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:43:08.696286 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696268 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.696342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696291 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.696342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696319 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.696342 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696324 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.696440 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696387 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.696440 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.696395 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="054a0ab6-9818-4f2c-97ad-d82a1b5493ac" containerName="istio-proxy" Jul 09 12:43:08.699602 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.699586 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.702861 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.702835 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 12:43:08.703030 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.702918 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4861fa-openshift-default-dockercfg-x5cdw\"" Jul 09 12:43:08.712349 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.712322 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:43:08.736413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736612 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmdq\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736612 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736461 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736612 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.736863 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.736814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmdq\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838419 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838583 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838600 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838838 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838709 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.838986 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.838962 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.839226 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.839193 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.839528 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.839241 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.840916 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.840891 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.841135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.841114 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.848394 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.848357 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:08.848523 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:08.848475 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmdq\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq\") pod \"e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:09.011490 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:09.011386 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:09.168519 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:09.168489 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:43:09.171020 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:43:09.170967 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a99ab9_445b_48ea_9ef5_d527af9fba29.slice/crio-70a8991cbe083bdfa0041a22e966b5949c13ad69a1611a61c589440ccc07480c WatchSource:0}: Error finding container 70a8991cbe083bdfa0041a22e966b5949c13ad69a1611a61c589440ccc07480c: Status 404 returned error can't find the container with id 70a8991cbe083bdfa0041a22e966b5949c13ad69a1611a61c589440ccc07480c Jul 09 12:43:09.173603 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:09.173572 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:09.173689 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:09.173649 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:09.173689 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:09.173685 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:10.094641 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:10.094600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerStarted","Data":"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e"} Jul 09 12:43:10.095122 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:10.094653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerStarted","Data":"70a8991cbe083bdfa0041a22e966b5949c13ad69a1611a61c589440ccc07480c"} Jul 09 12:43:10.116790 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:10.116728 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podStartSLOduration=2.116710689 podStartE2EDuration="2.116710689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:43:10.113785541 +0000 UTC m=+2424.876511076" watchObservedRunningTime="2026-07-09 12:43:10.116710689 +0000 UTC m=+2424.879436225" Jul 09 12:43:11.012219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:11.012176 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:11.017161 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:11.017133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:11.100398 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:11.100368 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:11.101477 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:11.101458 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:17.168769 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.168735 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:43:17.172968 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.172915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.175415 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.175393 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a2f2c0-openshift-default-dockercfg-b6hg5\"" Jul 09 12:43:17.184787 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.184758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:43:17.226855 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.226801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.226855 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.226863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227163 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227163 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqh6x\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227163 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227190 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227215 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227277 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.227401 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.227320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.328899 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.328857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.328899 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.328909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.328961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329041 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqh6x\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329104 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329394 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329420 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329486 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.329760 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.329736 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.330273 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.330251 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.331745 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.331710 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.331877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.331855 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.338405 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.338376 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.339093 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.339069 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqh6x\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x\") pod \"e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.485799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.485758 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:17.641762 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.641731 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:43:17.644469 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:43:17.644438 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac140b_b352_4db9_8655_a2abf48d782f.slice/crio-ca6f3e0354809cc9afa7437b5858870a70aee3a4d9128579ed3d72f535fdf6b3 WatchSource:0}: Error finding container ca6f3e0354809cc9afa7437b5858870a70aee3a4d9128579ed3d72f535fdf6b3: Status 404 returned error can't find the container with id ca6f3e0354809cc9afa7437b5858870a70aee3a4d9128579ed3d72f535fdf6b3 Jul 09 12:43:17.646757 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.646727 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:17.646835 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.646805 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:17.646879 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:17.646834 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:18.142545 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:18.142504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerStarted","Data":"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69"} Jul 09 12:43:18.142734 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:18.142571 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerStarted","Data":"ca6f3e0354809cc9afa7437b5858870a70aee3a4d9128579ed3d72f535fdf6b3"} Jul 09 12:43:18.163857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:18.163797 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podStartSLOduration=1.163775882 podStartE2EDuration="1.163775882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:43:18.161796728 +0000 UTC m=+2432.924522314" watchObservedRunningTime="2026-07-09 12:43:18.163775882 +0000 UTC m=+2432.926501419" Jul 09 12:43:18.486432 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:18.486398 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:18.491374 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:18.491338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:19.149031 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:19.149003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:19.150036 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:19.150013 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:27.142148 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:27.142103 2567 patch_prober.go:28] interesting pod/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 12:43:27.142549 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:27.142175 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 12:43:29.217545 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.217510 2567 generic.go:356] "Generic (PLEG): container finished" podID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerID="3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e" exitCode=0 Jul 09 12:43:29.218119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.217592 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerDied","Data":"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e"} Jul 09 12:43:29.218189 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.218177 2567 scope.go:117] "RemoveContainer" containerID="3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e" Jul 09 12:43:29.219221 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.219195 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:29.219320 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.219282 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:29.219382 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:29.219331 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:30.225016 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:30.224970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerStarted","Data":"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620"} Jul 09 12:43:31.012033 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:31.011981 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:31.012470 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:31.012437 2567 patch_prober.go:28] interesting pod/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 12:43:31.012586 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:31.012512 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 12:43:32.012521 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:32.012470 2567 patch_prober.go:28] interesting pod/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 12:43:32.013078 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:32.012541 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 12:43:33.012270 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:33.012196 2567 patch_prober.go:28] interesting pod/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 12:43:33.012473 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:33.012286 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 12:43:34.012776 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:34.012739 2567 patch_prober.go:28] interesting pod/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 12:43:34.013173 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:34.012804 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 12:43:35.016603 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:35.016564 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:35.017141 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:35.017120 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:35.018048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:35.018025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:43:35.192117 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:35.192080 2567 patch_prober.go:28] interesting pod/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 12:43:35.192308 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:35.192135 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 12:43:37.267584 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.267543 2567 generic.go:356] "Generic (PLEG): container finished" podID="caac140b-b352-4db9-8655-a2abf48d782f" containerID="d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69" exitCode=0 Jul 09 12:43:37.268139 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.267613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerDied","Data":"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69"} Jul 09 12:43:37.268139 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.268099 2567 scope.go:117] "RemoveContainer" containerID="d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69" Jul 09 12:43:37.269322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.269289 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:37.269433 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.269373 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:37.269433 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:37.269420 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:43:38.279328 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:38.279293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerStarted","Data":"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c"} Jul 09 12:43:38.487082 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:38.486578 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:38.487299 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:38.487139 2567 patch_prober.go:28] interesting pod/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 12:43:38.487299 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:38.487198 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 12:43:39.486346 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:39.486295 2567 patch_prober.go:28] interesting pod/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 12:43:39.486857 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:39.486374 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 12:43:40.487287 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:40.487248 2567 patch_prober.go:28] interesting pod/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 12:43:40.487895 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:40.487860 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 12:43:41.237213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.237181 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:43:41.240139 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.240121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.243678 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.243651 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4861fa\"/\"test-model-4861fa-kserve-self-signed-certs\"" Jul 09 12:43:41.243678 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.243662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4861fa\"/\"default-dockercfg-mcwxj\"" Jul 09 12:43:41.243872 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.243663 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4861fa\"/\"openshift-service-ca.crt\"" Jul 09 12:43:41.243872 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.243663 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4861fa\"/\"kube-root-ca.crt\"" Jul 09 12:43:41.249655 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.249629 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:43:41.306621 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.306799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.306799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.306799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754g5\" (UniqueName: \"kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.306799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306740 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.306799 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.306782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407369 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754g5\" (UniqueName: \"kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407567 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.407791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.407616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.408106 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.408073 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.408245 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.408121 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.408245 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.408209 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.410225 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.410192 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.410462 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.410444 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.423889 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.423865 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-754g5\" (UniqueName: \"kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5\") pod \"test-model-4861fa-kserve-669c8f89d9-n8sd6\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.487401 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.487292 2567 patch_prober.go:28] interesting pod/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 12:43:41.487401 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.487367 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 12:43:41.552037 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.552003 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:41.749101 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:41.749037 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:43:41.751499 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:43:41.751475 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958d23c1_5672_4f62_859e_027fc44ab8a3.slice/crio-41bcd6c07551ee40869900844e4c07b03e338a32cb29c04fe85c9e439669d4c4 WatchSource:0}: Error finding container 41bcd6c07551ee40869900844e4c07b03e338a32cb29c04fe85c9e439669d4c4: Status 404 returned error can't find the container with id 41bcd6c07551ee40869900844e4c07b03e338a32cb29c04fe85c9e439669d4c4 Jul 09 12:43:42.313112 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.313077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" event={"ID":"958d23c1-5672-4f62-859e-027fc44ab8a3","Type":"ContainerStarted","Data":"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975"} Jul 09 12:43:42.313112 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.313117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" event={"ID":"958d23c1-5672-4f62-859e-027fc44ab8a3","Type":"ContainerStarted","Data":"41bcd6c07551ee40869900844e4c07b03e338a32cb29c04fe85c9e439669d4c4"} Jul 09 12:43:42.331316 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.331264 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" podStartSLOduration=1.331244541 podStartE2EDuration="1.331244541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:43:42.329091609 +0000 UTC m=+2457.091817144" watchObservedRunningTime="2026-07-09 12:43:42.331244541 +0000 UTC m=+2457.093970076" Jul 09 12:43:42.489989 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.489924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:42.490394 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.490300 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:42.491174 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:42.491159 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:43:51.552534 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:51.552492 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:51.553190 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:51.552554 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:51.566223 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:51.566189 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:43:52.382687 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:43:52.382656 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:44:43.326876 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:43.326800 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:44:58.348337 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.348295 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" podUID="958d23c1-5672-4f62-859e-027fc44ab8a3" containerName="main" containerID="cri-o://41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975" gracePeriod=45 Jul 09 12:44:58.597784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.597761 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:44:58.674022 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.673922 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674022 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-754g5\" (UniqueName: \"kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674055 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674085 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674126 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home" (OuterVolumeSpecName: "home") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:44:58.674214 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674143 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm\") pod \"958d23c1-5672-4f62-859e-027fc44ab8a3\" (UID: \"958d23c1-5672-4f62-859e-027fc44ab8a3\") " Jul 09 12:44:58.674385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674273 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:44:58.674385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674329 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:44:58.674647 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674627 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-model-cache\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:58.674703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674654 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-tmp-dir\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:58.674703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.674668 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-home\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:58.676412 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.676389 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5" (OuterVolumeSpecName: "kube-api-access-754g5") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "kube-api-access-754g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:44:58.676509 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.676412 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm" (OuterVolumeSpecName: "dshm") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:44:58.676509 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.676427 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "958d23c1-5672-4f62-859e-027fc44ab8a3" (UID: "958d23c1-5672-4f62-859e-027fc44ab8a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 12:44:58.735433 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.735399 2567 generic.go:356] "Generic (PLEG): container finished" podID="958d23c1-5672-4f62-859e-027fc44ab8a3" containerID="41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975" exitCode=0 Jul 09 12:44:58.735575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.735450 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" Jul 09 12:44:58.735575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.735491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" event={"ID":"958d23c1-5672-4f62-859e-027fc44ab8a3","Type":"ContainerDied","Data":"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975"} Jul 09 12:44:58.735575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.735544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6" event={"ID":"958d23c1-5672-4f62-859e-027fc44ab8a3","Type":"ContainerDied","Data":"41bcd6c07551ee40869900844e4c07b03e338a32cb29c04fe85c9e439669d4c4"} Jul 09 12:44:58.735575 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.735567 2567 scope.go:117] "RemoveContainer" containerID="41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975" Jul 09 12:44:58.747765 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.747671 2567 scope.go:117] "RemoveContainer" containerID="41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975" Jul 09 12:44:58.748197 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:44:58.748174 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975\": container with ID starting with 41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975 not found: ID does not exist" containerID="41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975" Jul 09 12:44:58.748283 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.748205 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975"} err="failed to get container status \"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975\": rpc error: code = NotFound desc = could not find container \"41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975\": container with ID starting with 41c4d8a563bebd4ee900d55d8ff1fb83360c740bfd441f606f75bba5588f9975 not found: ID does not exist" Jul 09 12:44:58.758028 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.758007 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:44:58.761623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.761596 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4861fa/test-model-4861fa-kserve-669c8f89d9-n8sd6"] Jul 09 12:44:58.776041 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.776022 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d23c1-5672-4f62-859e-027fc44ab8a3-dshm\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:58.776041 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.776042 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-754g5\" (UniqueName: \"kubernetes.io/projected/958d23c1-5672-4f62-859e-027fc44ab8a3-kube-api-access-754g5\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:58.776160 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:58.776051 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d23c1-5672-4f62-859e-027fc44ab8a3-tls-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:44:59.848398 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:44:59.848368 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958d23c1-5672-4f62-859e-027fc44ab8a3" path="/var/lib/kubelet/pods/958d23c1-5672-4f62-859e-027fc44ab8a3/volumes" Jul 09 12:45:27.041708 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:27.041675 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:45:27.042339 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:27.041960 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" containerID="cri-o://77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620" gracePeriod=30 Jul 09 12:45:32.302556 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.302496 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:45:32.401178 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401149 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjmdq\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401178 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401182 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401219 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401259 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401289 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401417 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401399 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401628 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401447 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401628 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401455 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:32.401628 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401485 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401628 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401507 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket\") pod \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\" (UID: \"d4a99ab9-445b-48ea-9ef5-d527af9fba29\") " Jul 09 12:45:32.401829 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401636 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:32.401829 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401745 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:45:32.402124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.401876 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:32.402124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.402029 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.402124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.402045 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.402124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.402059 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.402124 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.402071 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.402389 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.402201 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:32.403673 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.403646 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq" (OuterVolumeSpecName: "kube-api-access-kjmdq") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "kube-api-access-kjmdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:45:32.403673 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.403652 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:45:32.404258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.404229 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:32.404258 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.404232 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4a99ab9-445b-48ea-9ef5-d527af9fba29" (UID: "d4a99ab9-445b-48ea-9ef5-d527af9fba29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:45:32.502807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.502773 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.502807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.502802 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmdq\" (UniqueName: \"kubernetes.io/projected/d4a99ab9-445b-48ea-9ef5-d527af9fba29-kube-api-access-kjmdq\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.502807 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.502813 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.503094 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.502821 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.503094 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.502829 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a99ab9-445b-48ea-9ef5-d527af9fba29-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:32.915838 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.915801 2567 generic.go:356] "Generic (PLEG): container finished" podID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerID="77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620" exitCode=0 Jul 09 12:45:32.916042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.915868 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" Jul 09 12:45:32.916042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.915918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerDied","Data":"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620"} Jul 09 12:45:32.916042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.915986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw" event={"ID":"d4a99ab9-445b-48ea-9ef5-d527af9fba29","Type":"ContainerDied","Data":"70a8991cbe083bdfa0041a22e966b5949c13ad69a1611a61c589440ccc07480c"} Jul 09 12:45:32.916042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.916010 2567 scope.go:117] "RemoveContainer" containerID="77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620" Jul 09 12:45:32.926249 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.926233 2567 scope.go:117] "RemoveContainer" containerID="3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e" Jul 09 12:45:32.934966 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.934917 2567 scope.go:117] "RemoveContainer" containerID="77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620" Jul 09 12:45:32.935225 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:45:32.935203 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620\": container with ID starting with 77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620 not found: ID does not exist" containerID="77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620" Jul 09 12:45:32.935282 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.935233 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620"} err="failed to get container status \"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620\": rpc error: code = NotFound desc = could not find container \"77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620\": container with ID starting with 77ef943f9ffe7561d2c4be111d5472e88ccb35ef90d8ed405623159b355cf620 not found: ID does not exist" Jul 09 12:45:32.935282 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.935251 2567 scope.go:117] "RemoveContainer" containerID="3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e" Jul 09 12:45:32.935471 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:45:32.935453 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e\": container with ID starting with 3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e not found: ID does not exist" containerID="3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e" Jul 09 12:45:32.935512 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.935476 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e"} err="failed to get container status \"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e\": rpc error: code = NotFound desc = could not find container \"3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e\": container with ID starting with 3569c1abbe303408071b2ba44d56d40f524be4b21b3b8b287891a47af4368a5e not found: ID does not exist" Jul 09 12:45:32.939361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.939339 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:45:32.945384 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:32.945365 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4861fa-openshift-default-9f774b9c7-j7nrw"] Jul 09 12:45:33.848367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:33.848329 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" path="/var/lib/kubelet/pods/d4a99ab9-445b-48ea-9ef5-d527af9fba29/volumes" Jul 09 12:45:34.340623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:34.340590 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:45:34.340881 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:34.340829 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" containerID="cri-o://bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c" gracePeriod=30 Jul 09 12:45:35.185197 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.185161 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:45:35.185957 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.185909 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958d23c1-5672-4f62-859e-027fc44ab8a3" containerName="main" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186494 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="958d23c1-5672-4f62-859e-027fc44ab8a3" containerName="main" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186524 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186530 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186637 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="958d23c1-5672-4f62-859e-027fc44ab8a3" containerName="main" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186649 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:35.186684 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.186657 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:35.190156 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.190140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.192633 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.192614 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a7e56d-openshift-default-dockercfg-zqm8k\"" Jul 09 12:45:35.201322 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.201297 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:45:35.327517 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327517 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnszp\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327655 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327730 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.327755 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327749 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.328022 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.327784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429197 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429442 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429442 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429442 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnszp\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429558 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429565 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429607 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429651 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429692 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.429816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429783 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.430008 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.429826 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.430074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.430016 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.432000 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.431980 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.432239 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.432219 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.437279 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.437213 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.437438 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.437417 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnszp\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp\") pod \"e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.501877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.501832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:35.654441 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.654418 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:45:35.656324 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:45:35.656296 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189dd138_09c5_4b16_a287_462f3360db05.slice/crio-55bd27bc51ad7a672bfe9597aeb6ef0464405d33406590d8e80d94413049b545 WatchSource:0}: Error finding container 55bd27bc51ad7a672bfe9597aeb6ef0464405d33406590d8e80d94413049b545: Status 404 returned error can't find the container with id 55bd27bc51ad7a672bfe9597aeb6ef0464405d33406590d8e80d94413049b545 Jul 09 12:45:35.658793 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.658761 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:35.658895 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.658843 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:35.658895 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.658887 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:35.935950 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.935883 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerStarted","Data":"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934"} Jul 09 12:45:35.936121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.935974 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerStarted","Data":"55bd27bc51ad7a672bfe9597aeb6ef0464405d33406590d8e80d94413049b545"} Jul 09 12:45:35.956970 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:35.956850 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podStartSLOduration=0.956833733 podStartE2EDuration="956.833733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:45:35.95410598 +0000 UTC m=+2570.716831567" watchObservedRunningTime="2026-07-09 12:45:35.956833733 +0000 UTC m=+2570.719559269" Jul 09 12:45:36.502144 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:36.502120 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:36.507042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:36.507021 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:36.942416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:36.942391 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:36.943314 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:36.943297 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:39.607221 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.607168 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:45:39.676590 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676556 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676599 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676638 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqh6x\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676703 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676748 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.676785 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.677136 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676817 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.677136 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676849 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert\") pod \"caac140b-b352-4db9-8655-a2abf48d782f\" (UID: \"caac140b-b352-4db9-8655-a2abf48d782f\") " Jul 09 12:45:39.677136 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676892 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:39.677136 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.676985 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:39.677367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.677265 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.677367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.677285 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.677367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.677300 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:39.677367 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.677309 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:45:39.677558 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.677526 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data" (OuterVolumeSpecName: "istio-data") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:39.679256 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.679223 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token" (OuterVolumeSpecName: "istio-token") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:45:39.679387 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.679323 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:45:39.679676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.679647 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:45:39.679676 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.679652 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x" (OuterVolumeSpecName: "kube-api-access-vqh6x") pod "caac140b-b352-4db9-8655-a2abf48d782f" (UID: "caac140b-b352-4db9-8655-a2abf48d782f"). InnerVolumeSpecName "kube-api-access-vqh6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:45:39.777846 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777803 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.777846 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777834 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.777846 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777853 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac140b-b352-4db9-8655-a2abf48d782f-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.778121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777864 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.778121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777874 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac140b-b352-4db9-8655-a2abf48d782f-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.778121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777884 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac140b-b352-4db9-8655-a2abf48d782f-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.778121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.777894 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqh6x\" (UniqueName: \"kubernetes.io/projected/caac140b-b352-4db9-8655-a2abf48d782f-kube-api-access-vqh6x\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:45:39.959782 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.959747 2567 generic.go:356] "Generic (PLEG): container finished" podID="caac140b-b352-4db9-8655-a2abf48d782f" containerID="bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c" exitCode=0 Jul 09 12:45:39.959963 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.959806 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" Jul 09 12:45:39.959963 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.959838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerDied","Data":"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c"} Jul 09 12:45:39.959963 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.959899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9" event={"ID":"caac140b-b352-4db9-8655-a2abf48d782f","Type":"ContainerDied","Data":"ca6f3e0354809cc9afa7437b5858870a70aee3a4d9128579ed3d72f535fdf6b3"} Jul 09 12:45:39.959963 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.959938 2567 scope.go:117] "RemoveContainer" containerID="bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c" Jul 09 12:45:39.972101 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.971895 2567 scope.go:117] "RemoveContainer" containerID="d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69" Jul 09 12:45:39.977473 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.977449 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:45:39.981780 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.981758 2567 scope.go:117] "RemoveContainer" containerID="bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c" Jul 09 12:45:39.981877 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.981825 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a2f2c0-openshift-default-6c4f74ddb7-zc6m9"] Jul 09 12:45:39.982094 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:45:39.982074 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c\": container with ID starting with bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c not found: ID does not exist" containerID="bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c" Jul 09 12:45:39.982165 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.982106 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c"} err="failed to get container status \"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c\": rpc error: code = NotFound desc = could not find container \"bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c\": container with ID starting with bd4aa9a02b8b02a831630e39b2d05bb38e25a67acd410ac5e1da88d1cb46a27c not found: ID does not exist" Jul 09 12:45:39.982165 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.982129 2567 scope.go:117] "RemoveContainer" containerID="d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69" Jul 09 12:45:39.982365 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:45:39.982346 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69\": container with ID starting with d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69 not found: ID does not exist" containerID="d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69" Jul 09 12:45:39.982408 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:39.982372 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69"} err="failed to get container status \"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69\": rpc error: code = NotFound desc = could not find container \"d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69\": container with ID starting with d17aac565ab9556e27945b5b6b56083b5731032544468f4e56025bde8a924f69 not found: ID does not exist" Jul 09 12:45:41.848752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:41.848725 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caac140b-b352-4db9-8655-a2abf48d782f" path="/var/lib/kubelet/pods/caac140b-b352-4db9-8655-a2abf48d782f/volumes" Jul 09 12:45:43.096137 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096101 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:45:43.096784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096752 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:43.096784 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096774 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4a99ab9-445b-48ea-9ef5-d527af9fba29" containerName="istio-proxy" Jul 09 12:45:43.096995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096830 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:45:43.096995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096839 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:45:43.096995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096861 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:45:43.096995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096870 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:45:43.096995 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.096984 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:45:43.100862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.100841 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.103225 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.103205 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bd2897-openshift-default-dockercfg-9rjfd\"" Jul 09 12:45:43.110705 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.110684 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:45:43.208672 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.208865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.208865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.208865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.209048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.209048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208941 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hcv\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.209048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.208969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.209048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.209024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.209187 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.209065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310102 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310102 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hcv\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310717 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310717 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310512 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310717 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310552 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310876 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310739 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.310876 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.310814 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.311445 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.311415 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.312849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.312824 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.312989 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.312971 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.317949 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.317908 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.318590 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.318571 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hcv\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv\") pod \"e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.413577 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.413490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:43.556113 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.556092 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:45:43.558116 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:45:43.558090 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08146c4_cbe9_4114_80ed_c052ae6e799d.slice/crio-89fdade9a291d5de0d8e2be22b154f87b9d0c23aa5c490b111b649dec6c64555 WatchSource:0}: Error finding container 89fdade9a291d5de0d8e2be22b154f87b9d0c23aa5c490b111b649dec6c64555: Status 404 returned error can't find the container with id 89fdade9a291d5de0d8e2be22b154f87b9d0c23aa5c490b111b649dec6c64555 Jul 09 12:45:43.560409 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.560380 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:43.560487 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.560443 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:43.560487 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.560474 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:43.983500 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.983464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerStarted","Data":"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc"} Jul 09 12:45:43.983663 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:43.983522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerStarted","Data":"89fdade9a291d5de0d8e2be22b154f87b9d0c23aa5c490b111b649dec6c64555"} Jul 09 12:45:44.006612 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:44.006565 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podStartSLOduration=1.006548546 podStartE2EDuration="1.006548546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:45:44.004096076 +0000 UTC m=+2578.766821630" watchObservedRunningTime="2026-07-09 12:45:44.006548546 +0000 UTC m=+2578.769274081" Jul 09 12:45:44.414480 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:44.414380 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:44.419117 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:44.419092 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:44.988601 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:44.988573 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:44.989434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:44.989417 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:45:56.056380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.056348 2567 generic.go:356] "Generic (PLEG): container finished" podID="189dd138-09c5-4b16-a287-462f3360db05" containerID="577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934" exitCode=0 Jul 09 12:45:56.056882 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.056423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerDied","Data":"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934"} Jul 09 12:45:56.056990 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.056970 2567 scope.go:117] "RemoveContainer" containerID="577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934" Jul 09 12:45:56.058020 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.057994 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:56.058095 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.058065 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:56.058135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:56.058101 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:45:57.064241 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:57.064211 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerStarted","Data":"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec"} Jul 09 12:45:57.503250 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:57.503215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:45:57.503668 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:57.503573 2567 patch_prober.go:28] interesting pod/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 12:45:57.503846 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:57.503822 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 12:45:58.505372 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:58.505331 2567 patch_prober.go:28] interesting pod/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 12:45:58.506118 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:58.506072 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 12:45:59.502862 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:59.502812 2567 patch_prober.go:28] interesting pod/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 12:45:59.503121 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:45:59.502891 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 12:46:00.502739 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:00.502692 2567 patch_prober.go:28] interesting pod/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 12:46:00.503254 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:00.502763 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 12:46:01.502395 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:01.502350 2567 patch_prober.go:28] interesting pod/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 12:46:01.502598 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:01.502428 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 12:46:02.506391 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:02.506356 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:46:02.506870 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:02.506780 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:46:02.507530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:02.507512 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:46:05.125969 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.125940 2567 generic.go:356] "Generic (PLEG): container finished" podID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerID="2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc" exitCode=0 Jul 09 12:46:05.126363 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.126001 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerDied","Data":"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc"} Jul 09 12:46:05.126443 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.126427 2567 scope.go:117] "RemoveContainer" containerID="2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc" Jul 09 12:46:05.127440 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.127412 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:05.127522 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.127510 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:05.127559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.127547 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:05.330951 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.329963 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:46:05.330951 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:05.330187 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" containerID="cri-o://818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec" gracePeriod=30 Jul 09 12:46:06.133066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:06.132982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerStarted","Data":"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1"} Jul 09 12:46:06.413712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:06.413623 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:06.413885 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:06.413838 2567 patch_prober.go:28] interesting pod/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 12:46:06.413957 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:06.413914 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 12:46:07.414816 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:07.414775 2567 patch_prober.go:28] interesting pod/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 12:46:07.415291 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:07.414852 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 12:46:08.414653 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:08.414623 2567 patch_prober.go:28] interesting pod/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 12:46:08.414919 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:08.414894 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 12:46:09.414662 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:09.414621 2567 patch_prober.go:28] interesting pod/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 12:46:09.414848 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:09.414694 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 12:46:10.418020 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:10.417983 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:10.418462 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:10.418355 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:10.419195 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:10.419169 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:11.005494 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.005467 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:46:11.112413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112313 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112357 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnszp\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112384 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112413 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112410 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112429 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112447 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112502 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.112767 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112575 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs\") pod \"189dd138-09c5-4b16-a287-462f3360db05\" (UID: \"189dd138-09c5-4b16-a287-462f3360db05\") " Jul 09 12:46:11.113075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112997 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:11.113075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.112995 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:46:11.113274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.113139 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:11.113274 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.113277 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:11.113464 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.113389 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data" (OuterVolumeSpecName: "istio-data") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:11.114791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.114770 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token" (OuterVolumeSpecName: "istio-token") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:46:11.115074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.115051 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:46:11.115275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.115258 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp" (OuterVolumeSpecName: "kube-api-access-nnszp") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "kube-api-access-nnszp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:46:11.115326 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.115272 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "189dd138-09c5-4b16-a287-462f3360db05" (UID: "189dd138-09c5-4b16-a287-462f3360db05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:11.161321 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.161287 2567 generic.go:356] "Generic (PLEG): container finished" podID="189dd138-09c5-4b16-a287-462f3360db05" containerID="818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec" exitCode=0 Jul 09 12:46:11.161491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.161344 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" Jul 09 12:46:11.161491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.161372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerDied","Data":"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec"} Jul 09 12:46:11.161491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.161429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj" event={"ID":"189dd138-09c5-4b16-a287-462f3360db05","Type":"ContainerDied","Data":"55bd27bc51ad7a672bfe9597aeb6ef0464405d33406590d8e80d94413049b545"} Jul 09 12:46:11.161491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.161453 2567 scope.go:117] "RemoveContainer" containerID="818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec" Jul 09 12:46:11.171623 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.171605 2567 scope.go:117] "RemoveContainer" containerID="577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934" Jul 09 12:46:11.180580 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.180550 2567 scope.go:117] "RemoveContainer" containerID="818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec" Jul 09 12:46:11.181038 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:46:11.180917 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec\": container with ID starting with 818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec not found: ID does not exist" containerID="818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec" Jul 09 12:46:11.181133 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.181049 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec"} err="failed to get container status \"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec\": rpc error: code = NotFound desc = could not find container \"818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec\": container with ID starting with 818f1812dbf4cf7dae7f3ce719ee47ce52d957eb7c8cc8afe813d9a12f9afbec not found: ID does not exist" Jul 09 12:46:11.181133 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.181067 2567 scope.go:117] "RemoveContainer" containerID="577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934" Jul 09 12:46:11.181374 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:46:11.181356 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934\": container with ID starting with 577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934 not found: ID does not exist" containerID="577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934" Jul 09 12:46:11.181429 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.181379 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934"} err="failed to get container status \"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934\": rpc error: code = NotFound desc = could not find container \"577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934\": container with ID starting with 577545054c8a59286e19792d95b4c0666e75002b400584bd045c8c3ecadac934 not found: ID does not exist" Jul 09 12:46:11.183271 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.183249 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:46:11.186914 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.186890 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a7e56d-openshift-default-64c889486c-ffjxj"] Jul 09 12:46:11.213798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213767 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.213798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213793 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.213798 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213802 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnszp\" (UniqueName: \"kubernetes.io/projected/189dd138-09c5-4b16-a287-462f3360db05-kube-api-access-nnszp\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213812 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189dd138-09c5-4b16-a287-462f3360db05-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213824 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213835 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213854 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189dd138-09c5-4b16-a287-462f3360db05-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213862 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.214074 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.213869 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189dd138-09c5-4b16-a287-462f3360db05-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:11.848664 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:11.848623 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189dd138-09c5-4b16-a287-462f3360db05" path="/var/lib/kubelet/pods/189dd138-09c5-4b16-a287-462f3360db05/volumes" Jul 09 12:46:12.304910 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:12.304874 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:46:13.136908 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.136875 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:46:13.137390 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.137375 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:13.137444 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.137391 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:13.137486 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.137472 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:13.137486 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.137479 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="caac140b-b352-4db9-8655-a2abf48d782f" containerName="istio-proxy" Jul 09 12:46:13.137559 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.137487 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:13.140760 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.140739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.143218 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.143195 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e28f57-openshift-default-dockercfg-bt9zc\"" Jul 09 12:46:13.152232 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.152204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:46:13.174477 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.174438 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" containerID="cri-o://d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1" gracePeriod=30 Jul 09 12:46:13.230474 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230474 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmxg\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230861 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.230861 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.230759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332551 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmxg\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332730 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.332791 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332761 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.333048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.332850 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.333048 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.333035 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.333319 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.333300 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.335315 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.335297 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.335631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.335611 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.340078 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.340055 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.340266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.340246 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmxg\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg\") pod \"e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.451779 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.451746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:13.597061 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.597038 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:46:13.599399 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:46:13.599368 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ab5ae_1f70_4247_ac79_c1d9b2a7e651.slice/crio-7f33382bb59dcfb66f99d6d795e4242b321427189038361c44cb78f8c7e089b0 WatchSource:0}: Error finding container 7f33382bb59dcfb66f99d6d795e4242b321427189038361c44cb78f8c7e089b0: Status 404 returned error can't find the container with id 7f33382bb59dcfb66f99d6d795e4242b321427189038361c44cb78f8c7e089b0 Jul 09 12:46:13.601723 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.601693 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:13.601833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.601776 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:13.601833 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:13.601820 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:14.183447 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:14.183409 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerStarted","Data":"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3"} Jul 09 12:46:14.183981 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:14.183458 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerStarted","Data":"7f33382bb59dcfb66f99d6d795e4242b321427189038361c44cb78f8c7e089b0"} Jul 09 12:46:14.202355 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:14.202292 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podStartSLOduration=1.202272657 podStartE2EDuration="1.202272657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:46:14.200278115 +0000 UTC m=+2608.963003692" watchObservedRunningTime="2026-07-09 12:46:14.202272657 +0000 UTC m=+2608.964998193" Jul 09 12:46:14.452842 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:14.452805 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:14.458028 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:14.458005 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:15.188814 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:15.188785 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:15.189710 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:15.189691 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:18.441011 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.440988 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:18.487416 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487380 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487624 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487453 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48hcv\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487624 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487624 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487534 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487624 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487586 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487624 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487621 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487661 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487685 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.487886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487725 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:18.487886 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487753 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy\") pod \"d08146c4-cbe9-4114-80ed-c052ae6e799d\" (UID: \"d08146c4-cbe9-4114-80ed-c052ae6e799d\") " Jul 09 12:46:18.488116 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.487885 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:18.488182 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.488133 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.488182 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.488152 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.488182 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.488164 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:18.488491 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.488437 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:46:18.489039 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.489009 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:18.490479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.490438 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:46:18.490479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.490444 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv" (OuterVolumeSpecName: "kube-api-access-48hcv") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "kube-api-access-48hcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:46:18.490479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.490465 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:46:18.490664 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.490584 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d08146c4-cbe9-4114-80ed-c052ae6e799d" (UID: "d08146c4-cbe9-4114-80ed-c052ae6e799d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:46:18.588712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588672 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hcv\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-kube-api-access-48hcv\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.588712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588706 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.588712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588717 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.588712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588726 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.589023 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588734 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08146c4-cbe9-4114-80ed-c052ae6e799d-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.589023 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588741 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:18.589023 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:18.588750 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08146c4-cbe9-4114-80ed-c052ae6e799d-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:46:19.211903 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.211868 2567 generic.go:356] "Generic (PLEG): container finished" podID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerID="d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1" exitCode=0 Jul 09 12:46:19.212105 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.211917 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" Jul 09 12:46:19.212105 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.211956 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerDied","Data":"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1"} Jul 09 12:46:19.212105 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.212013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp" event={"ID":"d08146c4-cbe9-4114-80ed-c052ae6e799d","Type":"ContainerDied","Data":"89fdade9a291d5de0d8e2be22b154f87b9d0c23aa5c490b111b649dec6c64555"} Jul 09 12:46:19.212105 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.212035 2567 scope.go:117] "RemoveContainer" containerID="d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1" Jul 09 12:46:19.221733 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.221713 2567 scope.go:117] "RemoveContainer" containerID="2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc" Jul 09 12:46:19.230205 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.230183 2567 scope.go:117] "RemoveContainer" containerID="d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1" Jul 09 12:46:19.230471 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:46:19.230453 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1\": container with ID starting with d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1 not found: ID does not exist" containerID="d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1" Jul 09 12:46:19.230543 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.230481 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1"} err="failed to get container status \"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1\": rpc error: code = NotFound desc = could not find container \"d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1\": container with ID starting with d9061b20f2f79e76b9074c95575b213634f129ffdd621987106f3cff36a680c1 not found: ID does not exist" Jul 09 12:46:19.230543 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.230500 2567 scope.go:117] "RemoveContainer" containerID="2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc" Jul 09 12:46:19.230737 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:46:19.230716 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc\": container with ID starting with 2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc not found: ID does not exist" containerID="2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc" Jul 09 12:46:19.230782 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.230745 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc"} err="failed to get container status \"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc\": rpc error: code = NotFound desc = could not find container \"2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc\": container with ID starting with 2b5b573a5f53437da1945c03c1751cd032e7808f667080c96ec016d8904c44cc not found: ID does not exist" Jul 09 12:46:19.233659 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.233635 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:46:19.239344 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.239324 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bd2897-openshift-default-864bf47d49-jkmxp"] Jul 09 12:46:19.848502 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:19.848467 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" path="/var/lib/kubelet/pods/d08146c4-cbe9-4114-80ed-c052ae6e799d/volumes" Jul 09 12:46:21.162096 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162066 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:46:21.162716 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162694 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:21.162853 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162835 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="189dd138-09c5-4b16-a287-462f3360db05" containerName="istio-proxy" Jul 09 12:46:21.162918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162894 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.162918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162905 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.163066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162952 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.163066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.162961 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.163167 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.163094 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.163167 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.163112 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08146c4-cbe9-4114-80ed-c052ae6e799d" containerName="istio-proxy" Jul 09 12:46:21.165692 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.165666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.168172 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.168151 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e490e6-openshift-default-dockercfg-hrthp\"" Jul 09 12:46:21.179915 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.179889 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:46:21.214089 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214181 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87tfh\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214296 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214449 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.214591 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.214455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315452 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315647 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315647 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315647 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315824 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315824 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.315824 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87tfh\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316027 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316027 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316027 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.315887 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316027 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.316012 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316227 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.316160 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316290 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.316271 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.316646 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.316620 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.318242 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.318216 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.318352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.318335 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.324506 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.324474 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.324691 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.324671 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87tfh\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh\") pod \"e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.477977 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.477917 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:21.631537 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.631480 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:46:21.633484 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:46:21.633453 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3e66d5_0959_4c36_b080_2935b87c5fcc.slice/crio-1474501eca0ff5a3aa4711e2f1a849bd369784f4337f43ffe245692ec385118c WatchSource:0}: Error finding container 1474501eca0ff5a3aa4711e2f1a849bd369784f4337f43ffe245692ec385118c: Status 404 returned error can't find the container with id 1474501eca0ff5a3aa4711e2f1a849bd369784f4337f43ffe245692ec385118c Jul 09 12:46:21.635796 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.635763 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:21.635870 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.635835 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:21.635870 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:21.635865 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:22.230946 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:22.230901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerStarted","Data":"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020"} Jul 09 12:46:22.231361 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:22.230967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerStarted","Data":"1474501eca0ff5a3aa4711e2f1a849bd369784f4337f43ffe245692ec385118c"} Jul 09 12:46:22.277601 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:22.277547 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podStartSLOduration=1.277529471 podStartE2EDuration="1.277529471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:46:22.277128237 +0000 UTC m=+2617.039853773" watchObservedRunningTime="2026-07-09 12:46:22.277529471 +0000 UTC m=+2617.040255006" Jul 09 12:46:22.479025 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:22.478986 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:22.485465 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:22.485369 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:23.236909 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:23.236880 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:23.237814 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:23.237797 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:30.221167 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.219943 2567 patch_prober.go:28] interesting pod/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:34984->10.134.0.64:15021: read: connection reset by peer" start-of-body= Jul 09 12:46:30.221167 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.220034 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:34984->10.134.0.64:15021: read: connection reset by peer" Jul 09 12:46:30.297018 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.296979 2567 generic.go:356] "Generic (PLEG): container finished" podID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerID="2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3" exitCode=0 Jul 09 12:46:30.297213 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.297057 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerDied","Data":"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3"} Jul 09 12:46:30.297518 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.297501 2567 scope.go:117] "RemoveContainer" containerID="2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3" Jul 09 12:46:30.298560 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.298533 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:30.298631 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.298611 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:30.298670 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:30.298649 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:31.311952 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:31.311875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerStarted","Data":"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496"} Jul 09 12:46:31.452019 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:31.451982 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:31.452541 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:31.452516 2567 patch_prober.go:28] interesting pod/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 12:46:31.452646 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:31.452571 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 12:46:32.452484 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:32.452448 2567 patch_prober.go:28] interesting pod/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 12:46:32.452992 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:32.452963 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 12:46:33.452194 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:33.452153 2567 patch_prober.go:28] interesting pod/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 12:46:33.452477 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:33.452211 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 12:46:34.452325 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:34.452291 2567 patch_prober.go:28] interesting pod/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 12:46:34.452712 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:34.452358 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 12:46:35.456636 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:35.456602 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:35.457133 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:35.457055 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:35.457867 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:35.457837 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:46:37.355812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.355720 2567 generic.go:356] "Generic (PLEG): container finished" podID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerID="a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020" exitCode=0 Jul 09 12:46:37.355812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.355797 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerDied","Data":"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020"} Jul 09 12:46:37.356418 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.356371 2567 scope.go:117] "RemoveContainer" containerID="a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020" Jul 09 12:46:37.357455 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.357402 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:37.357519 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.357474 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:37.357564 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:37.357524 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 09 12:46:38.237564 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:38.237513 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:38.363313 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:38.363263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerStarted","Data":"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd"} Jul 09 12:46:38.478813 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:38.478654 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:38.480569 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:38.480231 2567 patch_prober.go:28] interesting pod/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 12:46:38.480569 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:38.480292 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 12:46:39.479224 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:39.479177 2567 patch_prober.go:28] interesting pod/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 12:46:39.479747 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:39.479252 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 12:46:40.478849 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:40.478808 2567 patch_prober.go:28] interesting pod/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 12:46:40.479196 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:40.479164 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 12:46:41.479226 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:41.479180 2567 patch_prober.go:28] interesting pod/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 12:46:41.479795 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:41.479259 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 12:46:42.479148 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:42.479106 2567 patch_prober.go:28] interesting pod/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 12:46:42.479421 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:42.479388 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 12:46:43.482249 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:43.482220 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:43.482708 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:43.482506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:46:43.483380 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:46:43.483348 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:49:40.842480 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:40.842448 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:49:40.843025 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:40.842692 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" containerID="cri-o://f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496" gracePeriod=30 Jul 09 12:49:46.105005 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.104912 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:49:46.123851 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.123820 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124021 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.123871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhmxg\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124021 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.123899 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124021 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.123959 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124028 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124056 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124114 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124156 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124218 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo\") pod \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\" (UID: \"169ab5ae-1f70-4247-ac79-c1d9b2a7e651\") " Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124436 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:49:46.124616 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124612 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.125043 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124719 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:46.125043 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124877 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:46.125043 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.124912 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:46.125802 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.125764 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data" (OuterVolumeSpecName: "istio-data") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:46.127238 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.127196 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg" (OuterVolumeSpecName: "kube-api-access-rhmxg") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "kube-api-access-rhmxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:49:46.127398 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.127340 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token" (OuterVolumeSpecName: "istio-token") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:49:46.127503 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.127419 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:46.127905 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.127883 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "169ab5ae-1f70-4247-ac79-c1d9b2a7e651" (UID: "169ab5ae-1f70-4247-ac79-c1d9b2a7e651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:49:46.225625 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225592 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225625 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225620 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmxg\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-kube-api-access-rhmxg\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225625 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225632 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225642 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225649 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225657 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225664 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.225865 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.225671 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169ab5ae-1f70-4247-ac79-c1d9b2a7e651-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:46.387845 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.387752 2567 generic.go:356] "Generic (PLEG): container finished" podID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerID="f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496" exitCode=0 Jul 09 12:49:46.387845 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.387812 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" Jul 09 12:49:46.387845 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.387834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerDied","Data":"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496"} Jul 09 12:49:46.388117 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.387916 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm" event={"ID":"169ab5ae-1f70-4247-ac79-c1d9b2a7e651","Type":"ContainerDied","Data":"7f33382bb59dcfb66f99d6d795e4242b321427189038361c44cb78f8c7e089b0"} Jul 09 12:49:46.388117 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.387957 2567 scope.go:117] "RemoveContainer" containerID="f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496" Jul 09 12:49:46.397900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.397881 2567 scope.go:117] "RemoveContainer" containerID="2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3" Jul 09 12:49:46.406422 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.406405 2567 scope.go:117] "RemoveContainer" containerID="f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496" Jul 09 12:49:46.406657 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:49:46.406637 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496\": container with ID starting with f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496 not found: ID does not exist" containerID="f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496" Jul 09 12:49:46.406715 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.406666 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496"} err="failed to get container status \"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496\": rpc error: code = NotFound desc = could not find container \"f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496\": container with ID starting with f3994a3a8d13a11ce1bb04b9ce31b99848bec7f478cf6b720dd8a86ebeaa4496 not found: ID does not exist" Jul 09 12:49:46.406715 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.406685 2567 scope.go:117] "RemoveContainer" containerID="2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3" Jul 09 12:49:46.406987 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:49:46.406965 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3\": container with ID starting with 2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3 not found: ID does not exist" containerID="2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3" Jul 09 12:49:46.407045 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.406994 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3"} err="failed to get container status \"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3\": rpc error: code = NotFound desc = could not find container \"2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3\": container with ID starting with 2a396f8d962395354982ff84afa9b9d295da3d435e47e2d23e5c57a91460d9c3 not found: ID does not exist" Jul 09 12:49:46.410188 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.410168 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:49:46.416060 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:46.416043 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e28f57-openshift-default-5bbb4c867-nsntm"] Jul 09 12:49:47.848392 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:47.848361 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" path="/var/lib/kubelet/pods/169ab5ae-1f70-4247-ac79-c1d9b2a7e651/volumes" Jul 09 12:49:47.971204 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:47.971174 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:49:47.971427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:47.971396 2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" containerID="cri-o://74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd" gracePeriod=30 Jul 09 12:49:48.764685 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:48.764656 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-668576498b-drsxm_0a822dd1-5700-4dfa-98d9-45280dce3eb6/authorino/0.log" Jul 09 12:49:53.223410 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.223383 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:49:53.294910 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.294833 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.294910 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.294873 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87tfh\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.294910 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.294900 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.294919 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295005 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295035 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295058 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295086 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295192 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295112 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data\") pod \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\" (UID: \"1e3e66d5-0959-4c36-b080-2935b87c5fcc\") " Jul 09 12:49:53.295496 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295203 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:53.295496 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295329 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:53.295496 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295379 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:53.295812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295537 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-certs\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.295812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295557 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-credential-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.295812 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295571 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-workload-socket\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.296001 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295814 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 12:49:53.296001 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.295990 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:53.297259 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.297231 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 12:49:53.297366 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.297278 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh" (OuterVolumeSpecName: "kube-api-access-87tfh") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "kube-api-access-87tfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:49:53.297776 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.297753 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 12:49:53.297776 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.297764 2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e3e66d5-0959-4c36-b080-2935b87c5fcc" (UID: "1e3e66d5-0959-4c36-b080-2935b87c5fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 12:49:53.316423 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.316405 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tsl65_1822ec44-47df-4500-8692-0c5950d2015f/manager/0.log" Jul 09 12:49:53.396766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396736 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-token\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.396766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396767 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istiod-ca-cert\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.396766 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396775 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-data\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.396998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396785 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-envoy\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.396998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396798 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87tfh\" (UniqueName: \"kubernetes.io/projected/1e3e66d5-0959-4c36-b080-2935b87c5fcc-kube-api-access-87tfh\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.396998 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.396809 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3e66d5-0959-4c36-b080-2935b87c5fcc-istio-podinfo\") on node \"ip-10-0-133-7.ec2.internal\" DevicePath \"\"" Jul 09 12:49:53.427786 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.427756 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f8c776fc-lsz4x_3c0a5098-1a51-4c25-9cb2-bb9c020bba03/manager/0.log" Jul 09 12:49:53.432664 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.432640 2567 generic.go:356] "Generic (PLEG): container finished" podID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerID="74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd" exitCode=0 Jul 09 12:49:53.432752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.432690 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" Jul 09 12:49:53.432752 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.432721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerDied","Data":"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd"} Jul 09 12:49:53.432828 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.432776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr" event={"ID":"1e3e66d5-0959-4c36-b080-2935b87c5fcc","Type":"ContainerDied","Data":"1474501eca0ff5a3aa4711e2f1a849bd369784f4337f43ffe245692ec385118c"} Jul 09 12:49:53.432828 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.432807 2567 scope.go:117] "RemoveContainer" containerID="74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd" Jul 09 12:49:53.442479 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.442462 2567 scope.go:117] "RemoveContainer" containerID="a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020" Jul 09 12:49:53.450673 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.450656 2567 scope.go:117] "RemoveContainer" containerID="74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd" Jul 09 12:49:53.450949 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:49:53.450916 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd\": container with ID starting with 74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd not found: ID does not exist" containerID="74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd" Jul 09 12:49:53.451003 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.450956 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd"} err="failed to get container status \"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd\": rpc error: code = NotFound desc = could not find container \"74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd\": container with ID starting with 74ae735a5c74358883d99c2e61682ec53593db0d7dbc29ee4e3b88df341bb6dd not found: ID does not exist" Jul 09 12:49:53.451003 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.450972 2567 scope.go:117] "RemoveContainer" containerID="a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020" Jul 09 12:49:53.451195 ip-10-0-133-7 kubenswrapper[2567]: E0709 12:49:53.451179 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020\": container with ID starting with a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020 not found: ID does not exist" containerID="a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020" Jul 09 12:49:53.451244 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.451197 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020"} err="failed to get container status \"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020\": rpc error: code = NotFound desc = could not find container \"a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020\": container with ID starting with a3f427adce5921c26189912520d2910b56ddebf0480ffd0a0c7ca39e43569020 not found: ID does not exist" Jul 09 12:49:53.454768 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.454747 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:49:53.458288 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.458269 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e490e6-openshift-default-5887767c67-pdzvr"] Jul 09 12:49:53.532594 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.532572 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mkwdm_afe8e85c-dd8f-4d1c-bbb9-3b665410c59f/server/0.log" Jul 09 12:49:53.645513 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.645445 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mq8hd_adb5a6bc-49b5-4c98-a343-a66acfce46b1/manager/0.log" Jul 09 12:49:53.754042 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.754020 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7989cf5-494ch_14f904c8-e496-4c30-815c-2d79bd502f9f/copy-manifests/0.log" Jul 09 12:49:53.766962 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.766897 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7989cf5-494ch_14f904c8-e496-4c30-815c-2d79bd502f9f/manager/0.log" Jul 09 12:49:53.849383 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:53.849354 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" path="/var/lib/kubelet/pods/1e3e66d5-0959-4c36-b080-2935b87c5fcc/volumes" Jul 09 12:49:55.270385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:55.270348 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-668576498b-drsxm_0a822dd1-5700-4dfa-98d9-45280dce3eb6/authorino/0.log" Jul 09 12:49:55.384096 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:55.384066 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bzlz8_afa3a7e9-edb6-4c55-97eb-45fcc3594b48/manager/0.log" Jul 09 12:49:55.495729 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:55.495695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j2vdf_69e00e53-6251-4776-a221-0e166d68a434/manager/0.log" Jul 09 12:49:55.725918 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:55.725890 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pd6qz_9e37f6d4-c593-40cc-9e14-2c3b4382ec81/registry-server/0.log" Jul 09 12:49:55.852409 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:55.852378 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ksbgd_0154982d-1ece-4dea-82a3-2f3bbe262038/manager/0.log" Jul 09 12:49:56.078683 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:56.078608 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-w7wsl_6c25f385-ce0e-4d6c-8895-f819c3392468/manager/0.log" Jul 09 12:49:56.762585 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:56.762555 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb4f5859d-lm6fr_01e03dea-1908-4135-86a5-43d6d29c1a68/kube-auth-proxy/0.log" Jul 09 12:49:56.996493 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:56.996452 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fkxp4_ae099e53-b928-4451-a8cf-5e03ede71b45/payload-pre-processing/0.log" Jul 09 12:49:57.107127 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:57.107051 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-r9j7f_722ee86a-4466-4618-90d1-1963090bf05e/payload-processing/0.log" Jul 09 12:49:57.552860 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:57.552829 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mqnfg_39a99598-b1ec-415b-90c0-3403271088d1/main/0.log" Jul 09 12:49:57.665089 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:57.665064 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6v745_2886505b-3611-4b2c-9760-73cb01d39960/main/0.log" Jul 09 12:49:57.888454 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:49:57.888382 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db99gdv_303f2f0a-6deb-4b86-aae3-ad306528f9e4/main/0.log" Jul 09 12:50:02.357637 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.357596 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jt4g/must-gather-hp8pn"] Jul 09 12:50:02.358072 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358057 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358074 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358085 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.358119 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358090 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.358219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358159 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.358219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358167 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358173 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358219 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358181 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.358338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358244 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358251 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3e66d5-0959-4c36-b080-2935b87c5fcc" containerName="istio-proxy" Jul 09 12:50:02.358338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358260 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.358338 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.358266 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="169ab5ae-1f70-4247-ac79-c1d9b2a7e651" containerName="istio-proxy" Jul 09 12:50:02.361788 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.361767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.364089 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.364068 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jt4g\"/\"openshift-service-ca.crt\"" Jul 09 12:50:02.364207 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.364088 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jt4g\"/\"kube-root-ca.crt\"" Jul 09 12:50:02.364871 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.364845 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jt4g\"/\"default-dockercfg-n6mxt\"" Jul 09 12:50:02.370806 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.370783 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jt4g/must-gather-hp8pn"] Jul 09 12:50:02.485152 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.485115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9593060-6778-4de3-a815-3cd2fdc1c705-must-gather-output\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.485152 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.485156 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrmw\" (UniqueName: \"kubernetes.io/projected/e9593060-6778-4de3-a815-3cd2fdc1c705-kube-api-access-9rrmw\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.586358 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.586311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9593060-6778-4de3-a815-3cd2fdc1c705-must-gather-output\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.586358 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.586357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrmw\" (UniqueName: \"kubernetes.io/projected/e9593060-6778-4de3-a815-3cd2fdc1c705-kube-api-access-9rrmw\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.586661 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.586638 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9593060-6778-4de3-a815-3cd2fdc1c705-must-gather-output\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.594093 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.594067 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrmw\" (UniqueName: \"kubernetes.io/projected/e9593060-6778-4de3-a815-3cd2fdc1c705-kube-api-access-9rrmw\") pod \"must-gather-hp8pn\" (UID: \"e9593060-6778-4de3-a815-3cd2fdc1c705\") " pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.672070 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.671982 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" Jul 09 12:50:02.814389 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.814357 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jt4g/must-gather-hp8pn"] Jul 09 12:50:02.817045 ip-10-0-133-7 kubenswrapper[2567]: W0709 12:50:02.817012 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9593060_6778_4de3_a815_3cd2fdc1c705.slice/crio-c6d854dbe3452b9df9006517e87046431602842c33ffaea517832dc66a810eb5 WatchSource:0}: Error finding container c6d854dbe3452b9df9006517e87046431602842c33ffaea517832dc66a810eb5: Status 404 returned error can't find the container with id c6d854dbe3452b9df9006517e87046431602842c33ffaea517832dc66a810eb5 Jul 09 12:50:02.819017 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:02.818999 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 12:50:03.484179 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:03.484147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" event={"ID":"e9593060-6778-4de3-a815-3cd2fdc1c705","Type":"ContainerStarted","Data":"c6d854dbe3452b9df9006517e87046431602842c33ffaea517832dc66a810eb5"} Jul 09 12:50:04.491573 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:04.491532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" event={"ID":"e9593060-6778-4de3-a815-3cd2fdc1c705","Type":"ContainerStarted","Data":"22df605d86af98c7514ffbb1ef38c54a42c13b1730d928f88df9e5b3eeb14a9c"} Jul 09 12:50:04.492066 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:04.491583 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" event={"ID":"e9593060-6778-4de3-a815-3cd2fdc1c705","Type":"ContainerStarted","Data":"f4320ce4f24e5d2f5a748979855d386d2887baee5a7f7984938683a6eaf65ccd"} Jul 09 12:50:04.511275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:04.509825 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jt4g/must-gather-hp8pn" podStartSLOduration=1.357485767 podStartE2EDuration="2.509803315s" totalImagesPullingTime="1.152317548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 12:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:50:04.508392881 +0000 UTC m=+2839.271118417" watchObservedRunningTime="2026-07-09 12:50:04.509803315 +0000 UTC m=+2839.272528852" Jul 09 12:50:05.520311 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:05.520282 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5cb4v_9519607a-8be8-4d08-839c-eb1c0ac80f01/global-pull-secret-syncer/0.log" Jul 09 12:50:05.728072 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:05.728041 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q9bb6_f5235ece-eae9-41ae-923f-642662767655/konnectivity-agent/0.log" Jul 09 12:50:05.790900 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:05.790776 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-7.ec2.internal_e2bff7cbf82451b4333adfc3db2c0112/haproxy/0.log" Jul 09 12:50:11.066964 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.065921 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-668576498b-drsxm_0a822dd1-5700-4dfa-98d9-45280dce3eb6/authorino/0.log" Jul 09 12:50:11.106625 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.106576 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bzlz8_afa3a7e9-edb6-4c55-97eb-45fcc3594b48/manager/0.log" Jul 09 12:50:11.138738 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.138682 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j2vdf_69e00e53-6251-4776-a221-0e166d68a434/manager/0.log" Jul 09 12:50:11.218137 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.218099 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pd6qz_9e37f6d4-c593-40cc-9e14-2c3b4382ec81/registry-server/0.log" Jul 09 12:50:11.395802 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.395713 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ksbgd_0154982d-1ece-4dea-82a3-2f3bbe262038/manager/0.log" Jul 09 12:50:11.494075 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:11.494041 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-w7wsl_6c25f385-ce0e-4d6c-8895-f819c3392468/manager/0.log" Jul 09 12:50:13.155434 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.155399 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/alertmanager/0.log" Jul 09 12:50:13.180974 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.180865 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/config-reloader/0.log" Jul 09 12:50:13.206953 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.206903 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/kube-rbac-proxy-web/0.log" Jul 09 12:50:13.232238 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.232183 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/kube-rbac-proxy/0.log" Jul 09 12:50:13.259432 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.259365 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/kube-rbac-proxy-metric/0.log" Jul 09 12:50:13.285138 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.285111 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/prom-label-proxy/0.log" Jul 09 12:50:13.314783 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.314754 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e7aac14-b869-4c50-8336-a74605822a30/init-config-reloader/0.log" Jul 09 12:50:13.514638 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.514608 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-f7cb7_97be61b3-733b-47de-ab1c-483d79194fe1/monitoring-plugin/0.log" Jul 09 12:50:13.644230 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.644158 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9v_2f7180d2-2f04-453e-bb5a-02693c15d8b4/node-exporter/0.log" Jul 09 12:50:13.667176 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.667150 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9v_2f7180d2-2f04-453e-bb5a-02693c15d8b4/kube-rbac-proxy/0.log" Jul 09 12:50:13.689866 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:13.689833 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9v_2f7180d2-2f04-453e-bb5a-02693c15d8b4/init-textfile/0.log" Jul 09 12:50:14.228761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.228736 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-8crd5_4c22707e-a1f3-4c72-8d80-d1bb97d984a8/prometheus-operator-admission-webhook/0.log" Jul 09 12:50:14.260028 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.259998 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c97df5865-2jzfc_0202d18d-d511-4820-a351-5e8a37a2cb1f/telemeter-client/0.log" Jul 09 12:50:14.282773 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.282747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c97df5865-2jzfc_0202d18d-d511-4820-a351-5e8a37a2cb1f/reload/0.log" Jul 09 12:50:14.307517 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.307463 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c97df5865-2jzfc_0202d18d-d511-4820-a351-5e8a37a2cb1f/kube-rbac-proxy/0.log" Jul 09 12:50:14.355149 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.355121 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/thanos-query/0.log" Jul 09 12:50:14.376883 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.376853 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/kube-rbac-proxy-web/0.log" Jul 09 12:50:14.400385 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.400347 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/kube-rbac-proxy/0.log" Jul 09 12:50:14.421473 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.421443 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/prom-label-proxy/0.log" Jul 09 12:50:14.442732 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.442702 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/kube-rbac-proxy-rules/0.log" Jul 09 12:50:14.466374 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:14.466348 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759bd7bdc4-9wt6j_6eedb07d-bb05-4f16-8c5d-0e6ef904568a/kube-rbac-proxy-metrics/0.log" Jul 09 12:50:15.591433 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.591392 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-ww6pm_a8b134f5-0b4d-4b51-8bfe-12c8bfc7a3fa/networking-console-plugin/0.log" Jul 09 12:50:15.655586 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.655546 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx"] Jul 09 12:50:15.661286 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.661253 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.674776 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.674747 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx"] Jul 09 12:50:15.750838 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.750792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-podres\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.751079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.750855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-lib-modules\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.751079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.750995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bm5w\" (UniqueName: \"kubernetes.io/projected/f44d17ff-2f40-42e9-a04d-dbf6153226c2-kube-api-access-2bm5w\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.751079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.751020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-proc\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.751079 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.751073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-sys\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.851971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-sys\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-podres\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852054 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-lib-modules\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852111 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-sys\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852184 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-lib-modules\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bm5w\" (UniqueName: \"kubernetes.io/projected/f44d17ff-2f40-42e9-a04d-dbf6153226c2-kube-api-access-2bm5w\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852352 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-proc\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852578 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852454 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-proc\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.852989 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.852972 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44d17ff-2f40-42e9-a04d-dbf6153226c2-podres\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.860761 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.860724 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bm5w\" (UniqueName: \"kubernetes.io/projected/f44d17ff-2f40-42e9-a04d-dbf6153226c2-kube-api-access-2bm5w\") pod \"perf-node-gather-daemonset-t2lrx\" (UID: \"f44d17ff-2f40-42e9-a04d-dbf6153226c2\") " pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:15.978838 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:15.978805 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:16.192300 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.191660 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx"] Jul 09 12:50:16.584944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.584886 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" event={"ID":"f44d17ff-2f40-42e9-a04d-dbf6153226c2","Type":"ContainerStarted","Data":"a5053c15a734894e81d494ee60c46cf0933779d5f409acd3a3481a650135e4a0"} Jul 09 12:50:16.585112 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.585063 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:16.585112 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.585091 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" event={"ID":"f44d17ff-2f40-42e9-a04d-dbf6153226c2","Type":"ContainerStarted","Data":"61911d101a94644f0b08be4cf828034a730a26a9e86f537545a3707687aa409a"} Jul 09 12:50:16.600744 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.600695 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" podStartSLOduration=1.6006794260000001 podStartE2EDuration="1.600679426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 12:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 12:50:16.599207523 +0000 UTC m=+2851.361933088" watchObservedRunningTime="2026-07-09 12:50:16.600679426 +0000 UTC m=+2851.363404960" Jul 09 12:50:16.801275 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.801243 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67d998dcf5-z5qr9_1b6d7696-dc84-45f7-aaa2-9ecdcd546a5c/console/0.log" Jul 09 12:50:16.836888 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:16.836803 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-xc2rq_9a5ee676-65ea-42ba-bc34-82449f7bbd4e/download-server/0.log" Jul 09 12:50:18.481464 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:18.481429 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkx95_4c69ba0e-f1cf-4c6e-902d-27afaecf1a49/dns/0.log" Jul 09 12:50:18.507343 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:18.507314 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkx95_4c69ba0e-f1cf-4c6e-902d-27afaecf1a49/kube-rbac-proxy/0.log" Jul 09 12:50:18.581178 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:18.581141 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bjdxm_12901fb7-1801-4666-a17b-cc1d15663480/dns-node-resolver/0.log" Jul 09 12:50:19.166703 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:19.166674 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cccb88cf4-djfnb_0d9f30c0-3f06-4644-92a5-6a09f74fe311/registry/0.log" Jul 09 12:50:19.191530 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:19.191484 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88pcx_153f22bc-d128-46ad-bb76-7ee70144b64b/node-ca/0.log" Jul 09 12:50:20.345041 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:20.345014 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb4f5859d-lm6fr_01e03dea-1908-4135-86a5-43d6d29c1a68/kube-auth-proxy/0.log" Jul 09 12:50:20.450076 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:20.450043 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fkxp4_ae099e53-b928-4451-a8cf-5e03ede71b45/payload-pre-processing/0.log" Jul 09 12:50:20.507695 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:20.507660 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-r9j7f_722ee86a-4466-4618-90d1-1963090bf05e/payload-processing/0.log" Jul 09 12:50:21.263177 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:21.263098 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qc2mn_9354067b-2812-4427-b351-ce4b19bb8b97/serve-healthcheck-canary/0.log" Jul 09 12:50:21.935427 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:21.935398 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxpnh_8fbd8b80-f0bf-464e-981b-bef2e22e9f89/kube-rbac-proxy/0.log" Jul 09 12:50:21.962316 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:21.962292 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxpnh_8fbd8b80-f0bf-464e-981b-bef2e22e9f89/exporter/0.log" Jul 09 12:50:21.986499 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:21.986467 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxpnh_8fbd8b80-f0bf-464e-981b-bef2e22e9f89/extractor/0.log" Jul 09 12:50:22.603666 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:22.603633 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jt4g/perf-node-gather-daemonset-t2lrx" Jul 09 12:50:24.273263 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.273215 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tsl65_1822ec44-47df-4500-8692-0c5950d2015f/manager/0.log" Jul 09 12:50:24.380177 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.380149 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f8c776fc-lsz4x_3c0a5098-1a51-4c25-9cb2-bb9c020bba03/manager/0.log" Jul 09 12:50:24.400970 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.400918 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mkwdm_afe8e85c-dd8f-4d1c-bbb9-3b665410c59f/server/0.log" Jul 09 12:50:24.476099 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.476010 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mq8hd_adb5a6bc-49b5-4c98-a343-a66acfce46b1/manager/0.log" Jul 09 12:50:24.556817 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.556743 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7989cf5-494ch_14f904c8-e496-4c30-815c-2d79bd502f9f/manager/0.log" Jul 09 12:50:24.577824 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:24.577795 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7989cf5-494ch_14f904c8-e496-4c30-815c-2d79bd502f9f/copy-manifests/0.log" Jul 09 12:50:26.393303 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:26.393233 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb4f5859d-lm6fr_01e03dea-1908-4135-86a5-43d6d29c1a68/kube-auth-proxy/0.log" Jul 09 12:50:26.484786 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:26.484747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fkxp4_ae099e53-b928-4451-a8cf-5e03ede71b45/payload-pre-processing/0.log" Jul 09 12:50:26.528944 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:26.528894 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-r9j7f_722ee86a-4466-4618-90d1-1963090bf05e/payload-processing/0.log" Jul 09 12:50:27.869266 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:27.869236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb4f5859d-lm6fr_01e03dea-1908-4135-86a5-43d6d29c1a68/kube-auth-proxy/0.log" Jul 09 12:50:27.966327 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:27.966294 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-fkxp4_ae099e53-b928-4451-a8cf-5e03ede71b45/payload-pre-processing/0.log" Jul 09 12:50:28.013135 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:28.013107 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-r9j7f_722ee86a-4466-4618-90d1-1963090bf05e/payload-processing/0.log" Jul 09 12:50:32.525680 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.525658 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/kube-multus-additional-cni-plugins/0.log" Jul 09 12:50:32.554639 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.554612 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/egress-router-binary-copy/0.log" Jul 09 12:50:32.579330 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.579308 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/cni-plugins/0.log" Jul 09 12:50:32.604853 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.604820 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/bond-cni-plugin/0.log" Jul 09 12:50:32.629735 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.629713 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/routeoverride-cni/0.log" Jul 09 12:50:32.651884 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.651862 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/whereabouts-cni-bincopy/0.log" Jul 09 12:50:32.672811 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:32.672791 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zv7s_db5b0a05-facd-4635-9817-50ca8be76cf5/whereabouts-cni/0.log" Jul 09 12:50:33.117099 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:33.117063 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vs42f_e2107a85-7439-4282-b45c-2f67d06a9489/kube-multus/0.log" Jul 09 12:50:33.139967 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:33.139939 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jwdfr_c7547c86-a7e4-4555-9aa2-08919e5843b8/network-metrics-daemon/0.log" Jul 09 12:50:33.158415 ip-10-0-133-7 kubenswrapper[2567]: I0709 12:50:33.158373 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jwdfr_c7547c86-a7e4-4555-9aa2-08919e5843b8/kube-rbac-proxy/0.log"