Jul 28 19:36:20.436965 ip-10-0-132-248 systemd[1]: Starting Kubernetes Kubelet... Jul 28 19:36:20.882216 ip-10-0-132-248 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:36:20.882216 ip-10-0-132-248 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 19:36:20.882216 ip-10-0-132-248 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:36:20.882216 ip-10-0-132-248 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 19:36:20.882216 ip-10-0-132-248 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:36:20.883396 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.883312 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 19:36:20.889438 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889422 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889439 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889444 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889447 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889450 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889453 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889456 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889459 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889462 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889465 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889467 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889470 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889473 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889475 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889478 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889481 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.889478 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889486 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889489 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889492 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889495 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889497 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889500 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889503 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889505 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889508 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889510 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889513 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889515 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889518 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889520 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889523 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889526 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889529 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889532 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889536 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.889881 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889538 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889542 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889544 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889547 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889549 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889551 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889554 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889556 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889559 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889561 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889564 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889566 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889569 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889572 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889574 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889578 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889580 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889583 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889586 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889588 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.890329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889591 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889594 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889597 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889600 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889602 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889605 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889607 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889610 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889613 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889615 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889618 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889620 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889622 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889625 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889628 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889630 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889633 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889636 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889639 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889641 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.890857 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889644 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889647 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889652 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889654 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889668 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889671 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889674 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889677 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889680 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889683 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889686 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889689 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889692 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889695 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889697 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889700 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889703 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889706 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889709 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889711 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.891339 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889714 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889716 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889719 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.889721 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891233 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891240 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891243 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891246 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891249 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891253 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891255 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891258 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891260 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891263 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891265 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891268 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891271 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891273 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891276 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891280 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.891845 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891283 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891285 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891288 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891291 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891294 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891297 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891300 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891302 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891305 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891307 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891310 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891312 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891314 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891317 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891320 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891322 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891324 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891327 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891329 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891332 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.892381 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891334 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891337 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891339 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891342 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891345 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891348 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891350 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891353 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891355 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891358 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891360 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891364 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891366 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891369 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891371 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891374 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891376 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891379 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891381 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891383 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.892916 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891386 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891388 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891392 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891394 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891396 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891399 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891403 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891406 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891409 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891412 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891415 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891417 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891420 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891422 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891425 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891430 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891433 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891436 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891439 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.893411 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891441 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891443 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891447 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891451 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891454 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891457 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891460 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891462 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891465 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891467 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891470 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891472 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891475 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891477 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891480 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891482 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891484 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891487 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891489 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891491 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.893880 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891494 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891497 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891500 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891502 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891869 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891874 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891877 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891880 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891883 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891886 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891889 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891891 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891894 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891896 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891899 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891901 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891904 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891907 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891909 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.894382 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891912 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891914 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891917 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891919 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891922 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891924 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891927 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891929 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891932 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891934 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891937 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891939 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891943 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891947 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891950 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891953 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891956 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891959 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891962 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891964 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.894853 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891968 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891971 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891974 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891976 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891979 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891981 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.891983 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892000 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892006 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892009 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892013 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892017 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892020 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892022 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892025 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892027 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892029 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892032 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892034 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892037 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.895331 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892040 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892042 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892044 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892047 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892049 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892053 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892059 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892062 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892064 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892067 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892070 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892073 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892076 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892079 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892082 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892085 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892088 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892091 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892094 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892096 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.895823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892099 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892101 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892104 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892106 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892109 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892111 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892113 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892116 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892120 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892123 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892126 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892128 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892131 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892134 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892137 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892139 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892142 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892144 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892148 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892152 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.896307 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892155 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892158 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892160 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892163 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892238 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892248 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892260 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892266 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892270 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892273 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892277 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892281 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892284 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892287 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892290 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892294 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892297 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892299 2562 flags.go:64] FLAG: --cgroup-root="" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892302 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892305 2562 flags.go:64] FLAG: --client-ca-file="" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892308 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892311 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892315 2562 flags.go:64] FLAG: --cluster-domain="" Jul 28 19:36:20.896812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892318 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892321 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892324 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892328 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892332 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892335 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892338 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892342 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892345 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892348 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892351 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892354 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892357 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892361 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892364 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892367 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892370 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892372 2562 flags.go:64] FLAG: --enable-server="true" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892376 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892381 2562 flags.go:64] FLAG: --event-burst="100" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892384 2562 flags.go:64] FLAG: --event-qps="50" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892387 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892390 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892393 2562 flags.go:64] FLAG: --eviction-hard="" Jul 28 19:36:20.897372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892397 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892399 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892402 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892405 2562 flags.go:64] FLAG: --eviction-soft="" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892408 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892411 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892414 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892417 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892420 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892422 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892425 2562 flags.go:64] FLAG: --feature-gates="" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892429 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892432 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892435 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892438 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892443 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892446 2562 flags.go:64] FLAG: --help="false" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892449 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-248.ec2.internal" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892452 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892455 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892458 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892463 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892466 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892469 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 19:36:20.897969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892472 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892475 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892478 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892481 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892484 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892487 2562 flags.go:64] FLAG: --kube-reserved="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892490 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892493 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892496 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892499 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892502 2562 flags.go:64] FLAG: --lock-file="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892504 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892507 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892511 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892516 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892519 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892522 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892524 2562 flags.go:64] FLAG: --logging-format="text" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892527 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892531 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892533 2562 flags.go:64] FLAG: --manifest-url="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892536 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892540 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892545 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892549 2562 flags.go:64] FLAG: --max-pods="110" Jul 28 19:36:20.898672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892552 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892555 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892558 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892561 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892566 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892569 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892572 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892578 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892581 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892584 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892587 2562 flags.go:64] FLAG: --pod-cidr="" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892591 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892596 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892599 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892602 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892605 2562 flags.go:64] FLAG: --port="10250" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892608 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892611 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4189b6f86299697" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892614 2562 flags.go:64] FLAG: --qos-reserved="" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892617 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892619 2562 flags.go:64] FLAG: --register-node="true" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892622 2562 flags.go:64] FLAG: --register-with-taints="" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892626 2562 flags.go:64] FLAG: --registry-burst="10" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892629 2562 flags.go:64] FLAG: --registry-qps="5" Jul 28 19:36:20.899324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892632 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892634 2562 flags.go:64] FLAG: --reserved-memory="" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892638 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892641 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892644 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892647 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892651 2562 flags.go:64] FLAG: --runonce="false" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892654 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892669 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892673 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892676 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892679 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892682 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892686 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892689 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892692 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892695 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892698 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892701 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892704 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892707 2562 flags.go:64] FLAG: --system-cgroups="" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892710 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892716 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892718 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892721 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 19:36:20.899949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892725 2562 flags.go:64] FLAG: --tls-min-version="" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892728 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892731 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892734 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892737 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892740 2562 flags.go:64] FLAG: --v="2" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892744 2562 flags.go:64] FLAG: --version="false" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892748 2562 flags.go:64] FLAG: --vmodule="" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892751 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.892754 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892872 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892876 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892879 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892883 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892886 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892889 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892892 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892894 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892897 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892899 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892903 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.900567 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892906 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892908 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892911 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892914 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892916 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892919 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892922 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892931 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892934 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892937 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892939 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892942 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892946 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892950 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892953 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892955 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892958 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892961 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892963 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892966 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.901111 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892969 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892971 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892974 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892977 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892981 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892983 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892986 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892988 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892991 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892994 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.892996 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893000 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893004 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893007 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893009 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893012 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893015 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893018 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893020 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.901606 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893022 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893027 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893030 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893033 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893036 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893038 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893041 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893043 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893046 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893049 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893051 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893061 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893064 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893067 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893069 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893072 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893074 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893078 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893081 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893083 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.902118 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893086 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893089 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893091 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893094 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893097 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893100 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893103 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893105 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893108 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893110 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893113 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893115 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893118 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893120 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893124 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893127 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893129 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893132 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893134 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.902614 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893137 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893140 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893142 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893145 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893147 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893150 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893153 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893155 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893158 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.893160 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.893168 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.900915 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.900932 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.901784 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.901792 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 19:36:20.903080 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901868 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901873 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901877 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901880 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901882 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901885 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901888 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901890 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901894 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901896 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901899 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901902 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901905 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901907 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901910 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901913 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901916 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901919 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901921 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901924 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.903458 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901927 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901929 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901932 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901934 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901937 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901940 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901942 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901945 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901948 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901956 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901959 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901962 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901965 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901968 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901970 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901973 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901975 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901978 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901980 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901983 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.903951 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901986 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901988 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901991 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901993 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901996 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.901998 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902001 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902003 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902006 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902009 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902011 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902014 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902016 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902019 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902021 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902024 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902026 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902029 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902031 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902034 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.904429 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902046 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902050 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902059 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902061 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902064 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902067 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902069 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902072 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902075 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902077 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902080 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902082 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902085 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902087 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902091 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902095 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902098 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902101 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902104 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.904932 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902107 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902110 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902113 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902116 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902118 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902122 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902125 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902127 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902130 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902134 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902137 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902140 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902142 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902146 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902149 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902151 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902161 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902164 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902168 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.905463 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902172 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.902177 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902341 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902346 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902349 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902352 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902355 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902357 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902361 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902365 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902367 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902370 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902373 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902376 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:36:20.905948 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902379 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902382 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902390 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902392 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902395 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902397 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902400 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902402 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902405 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902407 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902410 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902412 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902414 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902417 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902420 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902423 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902431 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902434 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902436 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902439 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:36:20.906289 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902441 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902444 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902446 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902449 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902451 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902454 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902456 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902459 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902462 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902464 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902466 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902469 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902471 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902474 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902477 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902479 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902482 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902484 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902487 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902490 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:36:20.906785 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902492 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902495 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902497 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902500 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902502 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902505 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902508 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902510 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902513 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902521 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902524 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902527 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902529 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902531 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902534 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902536 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902539 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902541 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902544 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902546 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:36:20.907283 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902550 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902553 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902556 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902558 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902561 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902563 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902566 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902568 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902572 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902576 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902578 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902581 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902584 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902587 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902589 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902592 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902595 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902597 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902600 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902602 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:36:20.907823 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902604 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902607 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902617 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902620 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902622 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902625 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:20.902628 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.902633 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.902773 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.905121 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.906744 2562 server.go:1019] "Starting client certificate rotation" Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.906871 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 19:36:20.908306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.906904 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 19:36:20.909708 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.909692 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 19:36:20.916817 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.916771 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 19:36:20.926679 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.926646 2562 log.go:25] "Validated CRI v1 runtime API" Jul 28 19:36:20.930101 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.930087 2562 log.go:25] "Validated CRI v1 image API" Jul 28 19:36:20.930901 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.930886 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 19:36:20.934931 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.934908 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 19:36:20.935946 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.935926 2562 fs.go:135] Filesystem UUIDs: map[33ff59dd-e1c1-4c80-aeac-7e03296ce798:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e720e4ac-407b-4bea-ae1f-4119d00cb0f3:/dev/nvme0n1p4] Jul 28 19:36:20.935999 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.935946 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 19:36:20.940430 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.940328 2562 manager.go:216] Machine: {Timestamp:2026-07-28 19:36:20.939872359 +0000 UTC m=+0.375826302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098584 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b166bb48bd18f4d1293aa03448e4 SystemUUID:ec28b166-bb48-bd18-f4d1-293aa03448e4 BootID:6da9e55b-005f-411c-8f96-18c4274dccf5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bd:5e:e2:8d:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bd:5e:e2:8d:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:05:3d:83:73:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 19:36:20.940430 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.940424 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 19:36:20.940536 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.940526 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 19:36:20.944303 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.944282 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 19:36:20.944437 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.944306 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 19:36:20.945114 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.945104 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 19:36:20.945146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.945115 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 19:36:20.945146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.945124 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 19:36:20.945985 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.945975 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 19:36:20.947234 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.947224 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 19:36:20.949003 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.948992 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 28 19:36:20.949128 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949118 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 19:36:20.949160 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949149 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 28 19:36:20.949160 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949158 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 19:36:20.949217 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949174 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 19:36:20.949217 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949184 2562 kubelet.go:398] "Adding apiserver pod source" Jul 28 19:36:20.949217 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949193 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 19:36:20.949543 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949532 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 19:36:20.949588 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.949550 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 19:36:20.953559 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.953543 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 19:36:20.955821 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.955805 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 19:36:20.955876 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.955837 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 19:36:20.958559 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958545 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958569 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958575 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958582 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958592 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958599 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 19:36:20.958606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958608 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 19:36:20.958793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958614 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 19:36:20.958793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958620 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 19:36:20.958793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958626 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 19:36:20.958793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958639 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 19:36:20.958793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.958651 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 19:36:20.960487 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.960451 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 19:36:20.960487 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.960461 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 19:36:20.960601 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.960489 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 19:36:20.960601 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.960513 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 19:36:20.964517 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.964500 2562 server.go:1268] "Started kubelet" Jul 28 19:36:20.964627 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.964589 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 19:36:20.964846 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.964587 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 19:36:20.964928 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.964871 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 19:36:20.965374 ip-10-0-132-248 systemd[1]: Started Kubernetes Kubelet. Jul 28 19:36:20.967883 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.967334 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 28 19:36:20.968222 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.968206 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 19:36:20.971297 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.970168 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-248.ec2.internal.18c68b39677ab216 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-248.ec2.internal,UID:ip-10-0-132-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-248.ec2.internal,},FirstTimestamp:2026-07-28 19:36:20.964471318 +0000 UTC m=+0.400425261,LastTimestamp:2026-07-28 19:36:20.964471318 +0000 UTC m=+0.400425261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-248.ec2.internal,}" Jul 28 19:36:20.971726 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.971705 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 19:36:20.977124 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.977106 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 19:36:20.977565 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.977545 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 19:36:20.977638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.977562 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 19:36:20.977638 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.977629 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:20.978473 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978453 2562 factory.go:55] Registering systemd factory Jul 28 19:36:20.978554 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978487 2562 factory.go:223] Registration of the systemd container factory successfully Jul 28 19:36:20.978612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978560 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 19:36:20.978612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978577 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 19:36:20.978612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978586 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 19:36:20.978773 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978687 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 28 19:36:20.978773 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978696 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 28 19:36:20.978773 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978723 2562 factory.go:153] Registering CRI-O factory Jul 28 19:36:20.978773 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978743 2562 factory.go:223] Registration of the crio container factory successfully Jul 28 19:36:20.978920 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978785 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 19:36:20.978920 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978803 2562 factory.go:103] Registering Raw factory Jul 28 19:36:20.978920 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.978814 2562 manager.go:1196] Started watching for new ooms in manager Jul 28 19:36:20.979414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.979400 2562 manager.go:349] Starting recovery of all containers Jul 28 19:36:20.986130 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.986108 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8pqz7" Jul 28 19:36:20.987428 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.987401 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 19:36:20.987638 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:20.987614 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 19:36:20.987727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.987621 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 19:36:20.993282 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.993262 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8pqz7" Jul 28 19:36:20.995039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:20.995020 2562 manager.go:354] Recovery completed Jul 28 19:36:21.000958 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.000943 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.003426 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.003406 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.003511 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.003443 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.003511 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.003456 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.004145 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.004129 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 19:36:21.004145 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.004144 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 19:36:21.004273 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.004164 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 28 19:36:21.004273 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.004190 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.004998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.004983 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.005076 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.005013 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.005076 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.005029 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.005817 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.005804 2562 policy_none.go:49] "None policy: Start" Jul 28 19:36:21.005865 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.005823 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 19:36:21.005865 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.005834 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 19:36:21.006990 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.006978 2562 policy_none.go:47] "Start" Jul 28 19:36:21.077922 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.077894 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.155646 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.155621 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 19:36:21.155786 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.155688 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 19:36:21.155786 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.155723 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 19:36:21.155786 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.155766 2562 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 28 19:36:21.158228 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.158209 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:36:21.178794 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.178777 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.185246 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185232 2562 manager.go:337] "Starting Device Plugin manager" Jul 28 19:36:21.185339 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.185289 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 19:36:21.185339 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185304 2562 server.go:87] "Starting device plugin registration server" Jul 28 19:36:21.185596 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185582 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 19:36:21.185656 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185598 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 19:36:21.185727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185691 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 19:36:21.185779 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185774 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 19:36:21.185824 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.185782 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 19:36:21.187099 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.186693 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 19:36:21.187099 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.186754 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.192874 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.192856 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-248.ec2.internal\" not found" node="ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.256069 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.256036 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal"] Jul 28 19:36:21.256185 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.256124 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.257849 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.257833 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.257933 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.257862 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.257933 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.257873 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.260282 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.260269 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.260387 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.260372 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.260431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.260412 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.261036 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261025 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.261036 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261035 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.261163 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261064 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.261163 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261064 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.261163 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261077 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.261163 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.261084 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.263283 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.263267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.263366 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.263290 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.263953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.263938 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.264023 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.263962 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.264023 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.263973 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.280544 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.280520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.280630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.280549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65b35075ce250b6dbc5ab26eb37c0081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-248.ec2.internal\" (UID: \"65b35075ce250b6dbc5ab26eb37c0081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.280630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.280564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.286510 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.286491 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:36:21.287344 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.287329 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:36:21.287414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.287357 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:36:21.287414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.287372 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:36:21.287414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.287411 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.291403 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.291386 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-248.ec2.internal\" not found" node="ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.294959 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.294942 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-248.ec2.internal\" not found" node="ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.297692 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.297644 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.297692 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.297680 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-248.ec2.internal\": node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.328158 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.328137 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.381418 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.381418 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65b35075ce250b6dbc5ab26eb37c0081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-248.ec2.internal\" (UID: \"65b35075ce250b6dbc5ab26eb37c0081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.381596 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.381596 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.381596 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65b35075ce250b6dbc5ab26eb37c0081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-248.ec2.internal\" (UID: \"65b35075ce250b6dbc5ab26eb37c0081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.381596 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.381512 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53301e0f7e0fac1cf98483ed074ec54b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal\" (UID: \"53301e0f7e0fac1cf98483ed074ec54b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.428798 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.428711 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.529501 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.529460 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.593909 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.593870 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.599050 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.598876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:21.630735 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.630553 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.731162 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.731070 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.831635 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.831595 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.905463 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.905429 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 19:36:21.905919 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.905603 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 19:36:21.931707 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:21.931678 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:21.943323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.943301 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:36:21.999878 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.999843 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 19:31:21 +0000 UTC" deadline="2028-02-20 15:39:23.695962796 +0000 UTC" Jul 28 19:36:21.999878 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:21.999874 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13724h3m1.69609193s" Jul 28 19:36:22.032568 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.032537 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:22.123919 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:22.123882 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b35075ce250b6dbc5ab26eb37c0081.slice/crio-48acc527e62047847ac00a4d20f76906e084e77623df81f11d8554215c02e183 WatchSource:0}: Error finding container 48acc527e62047847ac00a4d20f76906e084e77623df81f11d8554215c02e183: Status 404 returned error can't find the container with id 48acc527e62047847ac00a4d20f76906e084e77623df81f11d8554215c02e183 Jul 28 19:36:22.126137 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:22.126090 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53301e0f7e0fac1cf98483ed074ec54b.slice/crio-402f893c9d7caae4f7ae80a7c06733771153baf2f0de7fbff5573442034f956a WatchSource:0}: Error finding container 402f893c9d7caae4f7ae80a7c06733771153baf2f0de7fbff5573442034f956a: Status 404 returned error can't find the container with id 402f893c9d7caae4f7ae80a7c06733771153baf2f0de7fbff5573442034f956a Jul 28 19:36:22.130201 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.130184 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 19:36:22.132987 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.132964 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:22.159502 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.159435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" event={"ID":"53301e0f7e0fac1cf98483ed074ec54b","Type":"ContainerStarted","Data":"402f893c9d7caae4f7ae80a7c06733771153baf2f0de7fbff5573442034f956a"} Jul 28 19:36:22.160521 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.160495 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" event={"ID":"65b35075ce250b6dbc5ab26eb37c0081","Type":"ContainerStarted","Data":"48acc527e62047847ac00a4d20f76906e084e77623df81f11d8554215c02e183"} Jul 28 19:36:22.236699 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.236592 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-248.ec2.internal\" not found" Jul 28 19:36:22.240444 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.240425 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:36:22.280261 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.280224 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" Jul 28 19:36:22.295749 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.295722 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 19:36:22.297262 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.297246 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" Jul 28 19:36:22.304787 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.304766 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 19:36:22.559098 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.559006 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:36:22.950676 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.950578 2562 apiserver.go:52] "Watching apiserver" Jul 28 19:36:22.961111 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.961066 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 19:36:22.961572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.961536 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-6g2s4","openshift-multus/multus-additional-cni-plugins-nlv45","openshift-multus/network-metrics-daemon-46x8p","openshift-network-operator/iptables-alerter-rqcnj","openshift-ovn-kubernetes/ovnkube-node-7ph2m","kube-system/konnectivity-agent-82zr4","kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th","openshift-cluster-node-tuning-operator/tuned-v7w4b","openshift-dns/node-resolver-flg66","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal","openshift-multus/multus-j8wd5","openshift-network-diagnostics/network-check-target-xfcxs","kube-system/global-pull-secret-syncer-kqqfs"] Jul 28 19:36:22.964270 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.964237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:22.966716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.966685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:22.966826 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.966767 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:22.967113 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.967041 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6cqm\"" Jul 28 19:36:22.967316 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.967301 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.967403 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.967367 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.970345 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.970322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:22.970452 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.970394 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:22.972751 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.972401 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:22.975329 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.975307 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.976264 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.975599 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dbm9w\"" Jul 28 19:36:22.976264 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.975611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.976264 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.975842 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 19:36:22.976264 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.976186 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.977695 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.977673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:22.978898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.978879 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 19:36:22.979468 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.979361 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.979628 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.979588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tgck\"" Jul 28 19:36:22.980116 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.980231 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 19:36:22.980652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980498 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 19:36:22.980652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980522 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 19:36:22.980652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980616 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.981210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980932 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 19:36:22.981210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.980991 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 19:36:22.981210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.981078 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 19:36:22.981210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.981171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7n6n\"" Jul 28 19:36:22.982827 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.982804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bwpl9\"" Jul 28 19:36:22.982939 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.982917 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:22.983110 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:22.983081 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:22.983441 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.983420 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.983800 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.983633 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.985738 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.985702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:22.988401 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.988184 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 19:36:22.988502 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.988440 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.988942 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.988801 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 19:36:22.988942 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.988826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989105 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9qnp\"" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysconfig\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-sys\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6jp\" (UniqueName: \"kubernetes.io/projected/76dacd0d-370b-4e98-bf35-8e3295f04c6a-kube-api-access-tm6jp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-script-lib\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxhj\" (UniqueName: \"kubernetes.io/projected/d96ca0b6-60e2-4e21-af8c-83ee35e99958-kube-api-access-bvxhj\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989160 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-systemd-units\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-ovn\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-netd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-modprobe-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjffc\" (UniqueName: \"kubernetes.io/projected/20acf618-f784-4e5a-b682-32ca663a056f-kube-api-access-pjffc\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-node-log\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drzx\" (UniqueName: \"kubernetes.io/projected/937e5351-5208-4756-9981-244c38e5b0d0-kube-api-access-4drzx\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:22.989547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-var-lib-kubelet\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-slash\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jpq\" (UniqueName: \"kubernetes.io/projected/023e36af-fadf-47f4-93de-719edd61b816-kube-api-access-89jpq\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-lib-modules\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-host\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-netns\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-kubernetes\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989669 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-systemd\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-var-lib-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-env-overrides\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/023e36af-fadf-47f4-93de-719edd61b816-hosts-file\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023e36af-fadf-47f4-93de-719edd61b816-tmp-dir\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937e5351-5208-4756-9981-244c38e5b0d0-iptables-alerter-script\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20acf618-f784-4e5a-b682-32ca663a056f-ovn-node-metrics-cert\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.990480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-kubelet\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.989977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-systemd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-bin\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937e5351-5208-4756-9981-244c38e5b0d0-host-slash\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-conf\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-run\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-config\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d1e8efb-4f43-486e-9192-11fd9a39c941-agent-certs\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d1e8efb-4f43-486e-9192-11fd9a39c941-konnectivity-ca\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-tuned\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-tmp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-etc-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.991294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.990512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-log-socket\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:22.992016 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.991439 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 19:36:22.992074 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.992016 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 19:36:22.992896 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.992409 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bq4s\"" Jul 28 19:36:22.992896 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.992718 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.992896 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.992883 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.993110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.992964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:22.994996 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.994978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j8wd5" Jul 28 19:36:22.995630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.995544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 19:36:22.997764 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.996751 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 19:36:22.997764 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.996827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8d7f\"" Jul 28 19:36:22.997764 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.997164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 19:36:22.997764 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.997690 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 19:36:22.998070 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:22.998050 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rld2f\"" Jul 28 19:36:23.016146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.016121 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6krt5" Jul 28 19:36:23.022475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.022442 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6krt5" Jul 28 19:36:23.080270 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.080231 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 19:36:23.091446 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-netns\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-systemd\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-cni-binary-copy\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-multus-certs\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-sys-fs\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-sys\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-netns\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-host\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-systemd\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-var-lib-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-sys\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-var-lib-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091742 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091832 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-tuned\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-cnibin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20acf618-f784-4e5a-b682-32ca663a056f-ovn-node-metrics-cert\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-systemd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-bin\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.091956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-systemd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-conf\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.091999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-bin\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-run\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-config\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-serviceca\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-k8s-cni-cncf-io\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysctl-conf\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092152 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-run\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-netns\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-os-release\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-etc-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092248 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-etc-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6jp\" (UniqueName: \"kubernetes.io/projected/76dacd0d-370b-4e98-bf35-8e3295f04c6a-kube-api-access-tm6jp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.092838 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-hostroot\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-kubelet-config\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-script-lib\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-systemd-units\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-netd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-var-lib-kubelet\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-multus\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-conf-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-multus-daemon-config\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-openvswitch\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-node-log\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-var-lib-kubelet\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-systemd-units\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-config\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-cni-netd\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.093638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-node-log\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drzx\" (UniqueName: \"kubernetes.io/projected/937e5351-5208-4756-9981-244c38e5b0d0-kube-api-access-4drzx\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092797 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-etc-kubernetes\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hcp\" (UniqueName: \"kubernetes.io/projected/9832ca4e-ceef-494c-a242-1782202615a4-kube-api-access-c5hcp\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rhk\" (UniqueName: \"kubernetes.io/projected/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kube-api-access-j4rhk\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-slash\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jpq\" (UniqueName: \"kubernetes.io/projected/023e36af-fadf-47f4-93de-719edd61b816-kube-api-access-89jpq\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-lib-modules\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-kubernetes\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-ovnkube-script-lib\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-kubelet\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-kubernetes\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.092949 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-slash\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-lib-modules\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqjd\" (UniqueName: \"kubernetes.io/projected/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-kube-api-access-2sqjd\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-socket-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.094619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-device-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-env-overrides\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/023e36af-fadf-47f4-93de-719edd61b816-hosts-file\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023e36af-fadf-47f4-93de-719edd61b816-tmp-dir\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937e5351-5208-4756-9981-244c38e5b0d0-iptables-alerter-script\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937e5351-5208-4756-9981-244c38e5b0d0-host-slash\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/023e36af-fadf-47f4-93de-719edd61b816-hosts-file\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-dbus\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-tmp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-system-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-kubelet\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/937e5351-5208-4756-9981-244c38e5b0d0-host-slash\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-registration-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d1e8efb-4f43-486e-9192-11fd9a39c941-agent-certs\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d1e8efb-4f43-486e-9192-11fd9a39c941-konnectivity-ca\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:23.095499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-socket-dir-parent\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093628 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023e36af-fadf-47f4-93de-719edd61b816-tmp-dir\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20acf618-f784-4e5a-b682-32ca663a056f-env-overrides\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093686 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-host-kubelet\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.093736 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-log-socket\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysconfig\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.093828 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:23.593805617 +0000 UTC m=+3.029759575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-sysconfig\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-system-cni-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-log-socket\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxhj\" (UniqueName: \"kubernetes.io/projected/d96ca0b6-60e2-4e21-af8c-83ee35e99958-kube-api-access-bvxhj\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-ovn\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.093981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-modprobe-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86hc\" (UniqueName: \"kubernetes.io/projected/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-kube-api-access-d86hc\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.096431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-os-release\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/937e5351-5208-4756-9981-244c38e5b0d0-iptables-alerter-script\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjffc\" (UniqueName: \"kubernetes.io/projected/20acf618-f784-4e5a-b682-32ca663a056f-kube-api-access-pjffc\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094125 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20acf618-f784-4e5a-b682-32ca663a056f-run-ovn\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-modprobe-d\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094220 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d1e8efb-4f43-486e-9192-11fd9a39c941-konnectivity-ca\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cnibin\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-host\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-bin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094378 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76dacd0d-370b-4e98-bf35-8e3295f04c6a-host\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.094449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.096620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20acf618-f784-4e5a-b682-32ca663a056f-ovn-node-metrics-cert\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.096921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-tmp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.097253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.097067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d1e8efb-4f43-486e-9192-11fd9a39c941-agent-certs\") pod \"konnectivity-agent-82zr4\" (UID: \"8d1e8efb-4f43-486e-9192-11fd9a39c941\") " pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:23.097850 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.097606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76dacd0d-370b-4e98-bf35-8e3295f04c6a-etc-tuned\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.099008 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.098988 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:23.099102 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.099014 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:23.099102 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.099029 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:23.099102 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.099086 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:23.59907138 +0000 UTC m=+3.035025327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:23.103850 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.103829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drzx\" (UniqueName: \"kubernetes.io/projected/937e5351-5208-4756-9981-244c38e5b0d0-kube-api-access-4drzx\") pod \"iptables-alerter-rqcnj\" (UID: \"937e5351-5208-4756-9981-244c38e5b0d0\") " pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.103989 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.103881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jpq\" (UniqueName: \"kubernetes.io/projected/023e36af-fadf-47f4-93de-719edd61b816-kube-api-access-89jpq\") pod \"node-resolver-flg66\" (UID: \"023e36af-fadf-47f4-93de-719edd61b816\") " pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.104251 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.104205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6jp\" (UniqueName: \"kubernetes.io/projected/76dacd0d-370b-4e98-bf35-8e3295f04c6a-kube-api-access-tm6jp\") pod \"tuned-v7w4b\" (UID: \"76dacd0d-370b-4e98-bf35-8e3295f04c6a\") " pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.105003 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.104975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjffc\" (UniqueName: \"kubernetes.io/projected/20acf618-f784-4e5a-b682-32ca663a056f-kube-api-access-pjffc\") pod \"ovnkube-node-7ph2m\" (UID: \"20acf618-f784-4e5a-b682-32ca663a056f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.106246 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.106224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxhj\" (UniqueName: \"kubernetes.io/projected/d96ca0b6-60e2-4e21-af8c-83ee35e99958-kube-api-access-bvxhj\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:23.195835 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.195793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-system-cni-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.195844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.195914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-system-cni-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.195964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86hc\" (UniqueName: \"kubernetes.io/projected/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-kube-api-access-d86hc\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.196025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.195990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-os-release\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cnibin\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-bin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-cni-binary-copy\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-multus-certs\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-bin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196194 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-sys-fs\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cnibin\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-host\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.196288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-cnibin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-serviceca\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-sys-fs\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-k8s-cni-cncf-io\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-k8s-cni-cncf-io\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-netns\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-host\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-os-release\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196473 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-hostroot\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-netns\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-cnibin\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-run-multus-certs\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-os-release\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-kubelet-config\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-hostroot\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.196902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-kubelet-config\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-os-release\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-multus\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-conf-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-cni-multus\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-multus-daemon-config\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-conf-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-etc-kubernetes\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hcp\" (UniqueName: \"kubernetes.io/projected/9832ca4e-ceef-494c-a242-1782202615a4-kube-api-access-c5hcp\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rhk\" (UniqueName: \"kubernetes.io/projected/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kube-api-access-j4rhk\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.196954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-kubelet\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqjd\" (UniqueName: \"kubernetes.io/projected/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-kube-api-access-2sqjd\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-socket-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-serviceca\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-device-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-dbus\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.197778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197134 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-etc-kubernetes\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197142 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-system-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.197211 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-socket-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-registration-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-system-cni-dir\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-device-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.197281 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:23.697262157 +0000 UTC m=+3.133216094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197314 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-registration-dir\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-host-var-lib-kubelet\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197383 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76938137-378f-4f48-99ab-dae7a5688d58-dbus\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-socket-dir-parent\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197523 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-etc-selinux\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.198567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-cni-binary-copy\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.199379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9832ca4e-ceef-494c-a242-1782202615a4-multus-daemon-config\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.199379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.197566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9832ca4e-ceef-494c-a242-1782202615a4-multus-socket-dir-parent\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.207259 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.207198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86hc\" (UniqueName: \"kubernetes.io/projected/8e7a676f-2b30-4b7a-ba6b-606a21ca6068-kube-api-access-d86hc\") pod \"node-ca-6g2s4\" (UID: \"8e7a676f-2b30-4b7a-ba6b-606a21ca6068\") " pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.207654 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.207632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hcp\" (UniqueName: \"kubernetes.io/projected/9832ca4e-ceef-494c-a242-1782202615a4-kube-api-access-c5hcp\") pod \"multus-j8wd5\" (UID: \"9832ca4e-ceef-494c-a242-1782202615a4\") " pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.208415 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.208394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqjd\" (UniqueName: \"kubernetes.io/projected/76415a4a-5a70-4b6d-bf8b-2a38258a58ce-kube-api-access-2sqjd\") pod \"multus-additional-cni-plugins-nlv45\" (UID: \"76415a4a-5a70-4b6d-bf8b-2a38258a58ce\") " pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.208572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.208527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rhk\" (UniqueName: \"kubernetes.io/projected/e8c8b01a-c6b4-46c4-95e8-f3515ae2e003-kube-api-access-j4rhk\") pod \"aws-ebs-csi-driver-node-pm7th\" (UID: \"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.231178 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.230994 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:36:23.278129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.278096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flg66" Jul 28 19:36:23.289132 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.289111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rqcnj" Jul 28 19:36:23.303257 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.303234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:23.308171 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.308150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:23.316869 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.316846 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" Jul 28 19:36:23.324735 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.324716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nlv45" Jul 28 19:36:23.333613 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.333594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6g2s4" Jul 28 19:36:23.342507 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.342485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" Jul 28 19:36:23.348481 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.348463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j8wd5" Jul 28 19:36:23.601023 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.600924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:23.601194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.601027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:23.601194 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601137 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:23.601194 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601146 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:23.601194 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601159 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:23.601194 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601170 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:23.601442 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601224 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.60120506 +0000 UTC m=+4.037158994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:23.601442 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.601245 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.601234153 +0000 UTC m=+4.037188088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:23.701925 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:23.701891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:23.702103 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.702060 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:23.702151 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:23.702134 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.702112676 +0000 UTC m=+4.138066608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:24.026552 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.026462 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:31:23 +0000 UTC" deadline="2028-01-08 06:01:50.602298933 +0000 UTC" Jul 28 19:36:24.026552 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.026504 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12682h25m26.575799271s" Jul 28 19:36:24.131962 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.131917 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dacd0d_370b_4e98_bf35_8e3295f04c6a.slice/crio-a5cf546805e4b87fb4a96fe0a2d9cf3a355d945f7f246437a1304e8f100daea7 WatchSource:0}: Error finding container a5cf546805e4b87fb4a96fe0a2d9cf3a355d945f7f246437a1304e8f100daea7: Status 404 returned error can't find the container with id a5cf546805e4b87fb4a96fe0a2d9cf3a355d945f7f246437a1304e8f100daea7 Jul 28 19:36:24.136716 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.136693 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9832ca4e_ceef_494c_a242_1782202615a4.slice/crio-e99942e16aa12dfea3c63fa81b767902640e822b23a080acc740f045f23df18f WatchSource:0}: Error finding container e99942e16aa12dfea3c63fa81b767902640e822b23a080acc740f045f23df18f: Status 404 returned error can't find the container with id e99942e16aa12dfea3c63fa81b767902640e822b23a080acc740f045f23df18f Jul 28 19:36:24.139584 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.139560 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7a676f_2b30_4b7a_ba6b_606a21ca6068.slice/crio-8be18afd0f524019d4867e0d1d7567b8688c504eb924b95230996ccf17fd109d WatchSource:0}: Error finding container 8be18afd0f524019d4867e0d1d7567b8688c504eb924b95230996ccf17fd109d: Status 404 returned error can't find the container with id 8be18afd0f524019d4867e0d1d7567b8688c504eb924b95230996ccf17fd109d Jul 28 19:36:24.140332 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.140251 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023e36af_fadf_47f4_93de_719edd61b816.slice/crio-e5c3a892f140e2909af295d59c109981353f680cd5ac1282bc8e9bbd822537ab WatchSource:0}: Error finding container e5c3a892f140e2909af295d59c109981353f680cd5ac1282bc8e9bbd822537ab: Status 404 returned error can't find the container with id e5c3a892f140e2909af295d59c109981353f680cd5ac1282bc8e9bbd822537ab Jul 28 19:36:24.141935 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.141902 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937e5351_5208_4756_9981_244c38e5b0d0.slice/crio-ae6b7100f6ef7551f9f9246e2a05aaf01e0d4e7880988cf25145b13e96aab686 WatchSource:0}: Error finding container ae6b7100f6ef7551f9f9246e2a05aaf01e0d4e7880988cf25145b13e96aab686: Status 404 returned error can't find the container with id ae6b7100f6ef7551f9f9246e2a05aaf01e0d4e7880988cf25145b13e96aab686 Jul 28 19:36:24.156065 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.156045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:24.156173 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.156154 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:24.156235 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.156190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:24.156271 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.156252 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:24.165543 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.165509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rqcnj" event={"ID":"937e5351-5208-4756-9981-244c38e5b0d0","Type":"ContainerStarted","Data":"ae6b7100f6ef7551f9f9246e2a05aaf01e0d4e7880988cf25145b13e96aab686"} Jul 28 19:36:24.165878 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.165846 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1e8efb_4f43_486e_9192_11fd9a39c941.slice/crio-0237f59e764c6323499fe75d939df861d048f1c8ff5fa8bdff368ab5dfb8950c WatchSource:0}: Error finding container 0237f59e764c6323499fe75d939df861d048f1c8ff5fa8bdff368ab5dfb8950c: Status 404 returned error can't find the container with id 0237f59e764c6323499fe75d939df861d048f1c8ff5fa8bdff368ab5dfb8950c Jul 28 19:36:24.167134 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.167107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flg66" event={"ID":"023e36af-fadf-47f4-93de-719edd61b816","Type":"ContainerStarted","Data":"e5c3a892f140e2909af295d59c109981353f680cd5ac1282bc8e9bbd822537ab"} Jul 28 19:36:24.167306 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.167282 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c8b01a_c6b4_46c4_95e8_f3515ae2e003.slice/crio-a7f8343029cf4d85fd97fef620a168a3356e9811dd353439bd8d93fb42101709 WatchSource:0}: Error finding container a7f8343029cf4d85fd97fef620a168a3356e9811dd353439bd8d93fb42101709: Status 404 returned error can't find the container with id a7f8343029cf4d85fd97fef620a168a3356e9811dd353439bd8d93fb42101709 Jul 28 19:36:24.167559 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.167451 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20acf618_f784_4e5a_b682_32ca663a056f.slice/crio-17a798f6cad0539b777273bb66d45833b56b053a57471052a488d6bb93c33ff0 WatchSource:0}: Error finding container 17a798f6cad0539b777273bb66d45833b56b053a57471052a488d6bb93c33ff0: Status 404 returned error can't find the container with id 17a798f6cad0539b777273bb66d45833b56b053a57471052a488d6bb93c33ff0 Jul 28 19:36:24.168545 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.168410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j8wd5" event={"ID":"9832ca4e-ceef-494c-a242-1782202615a4","Type":"ContainerStarted","Data":"e99942e16aa12dfea3c63fa81b767902640e822b23a080acc740f045f23df18f"} Jul 28 19:36:24.169269 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:24.169164 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76415a4a_5a70_4b6d_bf8b_2a38258a58ce.slice/crio-10eb437a4d04ea3a9b301130d4a4155eedb01ff4f22aad635461f152bff757ae WatchSource:0}: Error finding container 10eb437a4d04ea3a9b301130d4a4155eedb01ff4f22aad635461f152bff757ae: Status 404 returned error can't find the container with id 10eb437a4d04ea3a9b301130d4a4155eedb01ff4f22aad635461f152bff757ae Jul 28 19:36:24.170473 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.170451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6g2s4" event={"ID":"8e7a676f-2b30-4b7a-ba6b-606a21ca6068","Type":"ContainerStarted","Data":"8be18afd0f524019d4867e0d1d7567b8688c504eb924b95230996ccf17fd109d"} Jul 28 19:36:24.171500 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.171445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" event={"ID":"76dacd0d-370b-4e98-bf35-8e3295f04c6a","Type":"ContainerStarted","Data":"a5cf546805e4b87fb4a96fe0a2d9cf3a355d945f7f246437a1304e8f100daea7"} Jul 28 19:36:24.608902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.608805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:24.608902 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.608875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.608972 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.609027 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.609043 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.609057 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.609043 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:26.60902144 +0000 UTC m=+6.044975390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:24.609136 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.609102 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:26.609091518 +0000 UTC m=+6.045045449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:24.710379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:24.709751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:24.710379 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.709953 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:24.710379 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:24.710017 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:26.709999244 +0000 UTC m=+6.145953179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:25.027393 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.027246 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:31:23 +0000 UTC" deadline="2027-12-27 23:40:38.249988004 +0000 UTC" Jul 28 19:36:25.027393 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.027311 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12412h4m13.222681285s" Jul 28 19:36:25.158470 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.158424 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:25.158637 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:25.158579 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:25.182506 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.182468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerStarted","Data":"10eb437a4d04ea3a9b301130d4a4155eedb01ff4f22aad635461f152bff757ae"} Jul 28 19:36:25.193243 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.193194 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"17a798f6cad0539b777273bb66d45833b56b053a57471052a488d6bb93c33ff0"} Jul 28 19:36:25.202777 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.201528 2562 generic.go:356] "Generic (PLEG): container finished" podID="53301e0f7e0fac1cf98483ed074ec54b" containerID="9a8ca4ac80ab4d5bd3352650c2168c6c7ab4fd97c0ee1a82ab85da078506bf73" exitCode=0 Jul 28 19:36:25.202777 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.201638 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" event={"ID":"53301e0f7e0fac1cf98483ed074ec54b","Type":"ContainerDied","Data":"9a8ca4ac80ab4d5bd3352650c2168c6c7ab4fd97c0ee1a82ab85da078506bf73"} Jul 28 19:36:25.205597 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.205262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" event={"ID":"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003","Type":"ContainerStarted","Data":"a7f8343029cf4d85fd97fef620a168a3356e9811dd353439bd8d93fb42101709"} Jul 28 19:36:25.211401 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.211365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82zr4" event={"ID":"8d1e8efb-4f43-486e-9192-11fd9a39c941","Type":"ContainerStarted","Data":"0237f59e764c6323499fe75d939df861d048f1c8ff5fa8bdff368ab5dfb8950c"} Jul 28 19:36:25.232025 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:25.231220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" event={"ID":"65b35075ce250b6dbc5ab26eb37c0081","Type":"ContainerStarted","Data":"95654fb9ac85f9a6b783dd7f69b70e12e7ed36a833144b8663ae7ff0d30c55e1"} Jul 28 19:36:26.157053 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.156223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:26.157053 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.156387 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:26.157053 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.156578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:26.157053 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.156660 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:26.262690 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.262627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" event={"ID":"53301e0f7e0fac1cf98483ed074ec54b","Type":"ContainerStarted","Data":"ec3b38e3b6dbac5790132e356fd7c9d197d7c94a7660beb5494ba594ddc84628"} Jul 28 19:36:26.284917 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.284805 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-248.ec2.internal" podStartSLOduration=3.284782304 podStartE2EDuration="3.284782304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:25.248621173 +0000 UTC m=+4.684575133" watchObservedRunningTime="2026-07-28 19:36:26.284782304 +0000 UTC m=+5.720736259" Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.629983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.630047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630265 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630288 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630306 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630364 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:30.63034592 +0000 UTC m=+10.066299854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630821 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:26.630967 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.630876 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:30.630857881 +0000 UTC m=+10.066811818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:26.731997 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:26.731348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:26.731997 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.731534 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:26.731997 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:26.731604 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:30.731582752 +0000 UTC m=+10.167536689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:27.156750 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:27.156715 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:27.156937 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:27.156883 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:28.157403 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:28.156724 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:28.157403 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:28.156792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:28.157403 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:28.156893 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:28.157403 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:28.157352 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:29.160466 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:29.160433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:29.160980 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:29.160571 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:30.156321 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:30.156226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:30.156498 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.156404 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:30.156567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:30.156549 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:30.156723 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.156677 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:30.663748 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:30.663673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:30.663766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.663926 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.663945 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.663959 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.664022 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:38.664002409 +0000 UTC m=+18.099956346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:30.664131 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.664100 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:30.664361 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.664139 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:38.664125427 +0000 UTC m=+18.100079362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:30.765094 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:30.764425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:30.765094 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.764637 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:30.765094 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:30.764726 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:38.764689476 +0000 UTC m=+18.200643413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:31.158365 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:31.157793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:31.158365 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:31.157922 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:32.156481 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:32.156439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:32.156481 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:32.156491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:32.157119 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:32.156602 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:32.157119 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:32.156845 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:33.160789 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:33.160508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:33.160789 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:33.160654 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:34.156857 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:34.156824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:34.156857 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:34.156844 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:34.157061 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:34.156940 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:34.157061 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:34.157052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:35.156522 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:35.156471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:35.156984 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:35.156610 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:36.156885 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:36.156843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:36.157331 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:36.156841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:36.157331 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:36.156973 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:36.157331 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:36.157076 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:37.156476 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:37.156438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:37.156678 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:37.156574 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:38.156653 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:38.156613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:38.157179 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:38.156632 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:38.157179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.156744 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:38.157179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.156893 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:38.729991 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:38.729952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:38.730179 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:38.730020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:38.730179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730116 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:38.730179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730128 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:38.730179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730142 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:38.730179 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730151 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:38.730426 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730193 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:54.730170595 +0000 UTC m=+34.166124525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:38.730426 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.730214 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:54.730204748 +0000 UTC m=+34.166158681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:38.830581 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:38.830542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:38.830771 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.830732 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:38.830832 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:38.830814 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:54.830793147 +0000 UTC m=+34.266747093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:39.156887 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:39.156858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:39.157302 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:39.156983 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:40.156379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:40.156341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:40.156570 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:40.156341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:40.156570 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:40.156476 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:40.156570 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:40.156555 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:41.157266 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:41.157110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:41.157783 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:41.157680 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:42.156698 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:42.156629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:42.156698 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:42.156689 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:42.156931 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:42.156791 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:42.156990 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:42.156908 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:43.156915 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.156705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:43.157516 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:43.156996 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:43.299512 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.299319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"6d30408e8eb347b4930505151ca12fb18b199349207f09b72f67857e4a20719a"} Jul 28 19:36:43.299606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.299567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"9c8f1b6c3e8a066b15a7bccc89c45ce5fcb4cbb14acf1566694c9113102ca33f"} Jul 28 19:36:43.299689 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.299606 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"a13a644d19573da163f86ecf38db1951e81de7e70f382cb0e12b956a1757d4f8"} Jul 28 19:36:43.300808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.300782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flg66" event={"ID":"023e36af-fadf-47f4-93de-719edd61b816","Type":"ContainerStarted","Data":"f738ce8d4104d5a37789a3f56b55cf3ebcbce76d8ec48f7da694cb31fa8c91af"} Jul 28 19:36:43.302178 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.302154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" event={"ID":"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003","Type":"ContainerStarted","Data":"204987739ebe661b8d29377ea7a95ecc2380ef006a2512eb46da80f0d5657cc4"} Jul 28 19:36:43.304467 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.304187 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82zr4" event={"ID":"8d1e8efb-4f43-486e-9192-11fd9a39c941","Type":"ContainerStarted","Data":"aeb5654958d7d02ae2d5a118d45d948229863e5308e4c6cca041b24522267528"} Jul 28 19:36:43.306228 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.306209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j8wd5" event={"ID":"9832ca4e-ceef-494c-a242-1782202615a4","Type":"ContainerStarted","Data":"732cc7af70c49489cf840a9d9bb0b9d87c3665466093481f74079612c67df6f2"} Jul 28 19:36:43.307646 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.307625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6g2s4" event={"ID":"8e7a676f-2b30-4b7a-ba6b-606a21ca6068","Type":"ContainerStarted","Data":"9ab83aab87c5a452131975a0445ae48f924d3135f6b506790fcf9f34c06cc1ad"} Jul 28 19:36:43.309023 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.309000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" event={"ID":"76dacd0d-370b-4e98-bf35-8e3295f04c6a","Type":"ContainerStarted","Data":"385e93c87c8feef189f112801bbf2dbcf9f5cc8bc46cbf2d6743ccef6acd2f77"} Jul 28 19:36:43.310456 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.310437 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="9a8a77039556aed2bad8c4e0fb62454b05a428c9af534408921260ec57fd9f45" exitCode=0 Jul 28 19:36:43.310555 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.310464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"9a8a77039556aed2bad8c4e0fb62454b05a428c9af534408921260ec57fd9f45"} Jul 28 19:36:43.321055 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.321011 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-248.ec2.internal" podStartSLOduration=20.32099705 podStartE2EDuration="20.32099705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:26.285468708 +0000 UTC m=+5.721422666" watchObservedRunningTime="2026-07-28 19:36:43.32099705 +0000 UTC m=+22.756951006" Jul 28 19:36:43.340753 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.340718 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-flg66" podStartSLOduration=6.268982758 podStartE2EDuration="21.340706552s" totalImagesPullingTime="15.071723794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:43.320942312 +0000 UTC m=+22.756896265" watchObservedRunningTime="2026-07-28 19:36:43.340706552 +0000 UTC m=+22.776660495" Jul 28 19:36:43.340832 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.340806 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j8wd5" podStartSLOduration=2.519987626 podStartE2EDuration="21.340801799s" totalImagesPullingTime="18.820814173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:43.34024114 +0000 UTC m=+22.776195094" watchObservedRunningTime="2026-07-28 19:36:43.340801799 +0000 UTC m=+22.776755750" Jul 28 19:36:43.357194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.357144 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-82zr4" podStartSLOduration=6.288982673 podStartE2EDuration="21.357132971s" totalImagesPullingTime="15.068150298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:43.35691331 +0000 UTC m=+22.792867263" watchObservedRunningTime="2026-07-28 19:36:43.357132971 +0000 UTC m=+22.793086922" Jul 28 19:36:43.397052 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.397014 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6g2s4" podStartSLOduration=2.654576299 podStartE2EDuration="21.396999298s" totalImagesPullingTime="18.742422999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:43.396420516 +0000 UTC m=+22.832374467" watchObservedRunningTime="2026-07-28 19:36:43.396999298 +0000 UTC m=+22.832953276" Jul 28 19:36:43.422017 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:43.421978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v7w4b" podStartSLOduration=2.648112608 podStartE2EDuration="21.42196407s" totalImagesPullingTime="18.773851462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:43.421322751 +0000 UTC m=+22.857276723" watchObservedRunningTime="2026-07-28 19:36:43.42196407 +0000 UTC m=+22.857918060" Jul 28 19:36:44.156308 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.156281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:44.156515 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.156281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:44.156515 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:44.156390 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:44.156515 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:44.156445 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:44.316503 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.316477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rqcnj" event={"ID":"937e5351-5208-4756-9981-244c38e5b0d0","Type":"ContainerStarted","Data":"02740d18e9672f093730bb9106947030e414ceb58c466e833298880b5fb87975"} Jul 28 19:36:44.319127 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.319094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"4dfaa33b680a1c8dcce3e80b3b6efc8b0fb4f1f54869cf103fbe155293548a0d"} Jul 28 19:36:44.319232 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.319189 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"0399bdc7b7e4c2cef38ab845ff90baedfdfe59cdcb5966ddd6e476e8d11b052b"} Jul 28 19:36:44.319232 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.319228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"4a74bb09c89af022bc51a8af0583719c2adff0950b39d73de1c9bc0352f8e23b"} Jul 28 19:36:44.337380 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:44.337354 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 19:36:45.157000 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:45.156968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:45.157299 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:45.157093 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:45.199246 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:45.199158 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T19:36:44.337376332Z","UUID":"fafd78cd-b825-44c8-9c26-9ad75a94753a","Handler":null,"Name":"","Endpoint":""} Jul 28 19:36:45.200631 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:45.200614 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 19:36:45.200733 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:45.200648 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 19:36:45.324088 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:45.324050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" event={"ID":"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003","Type":"ContainerStarted","Data":"1391b35da9d50566caa547752a60a01799e6ed1d3a3d453f91c3dffda0a2b7af"} Jul 28 19:36:46.156190 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.156148 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:46.156465 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.156148 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:46.156465 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:46.156330 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:46.156465 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:46.156378 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:46.328829 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.328792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" event={"ID":"e8c8b01a-c6b4-46c4-95e8-f3515ae2e003","Type":"ContainerStarted","Data":"c814a920ed7ece78d095b062fc237bc22eb3aa09a270352bc0aa6233a2cd4321"} Jul 28 19:36:46.332708 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.332682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"0ce4a2ce6f799ef7d58704df58f74e15e6216d06113ac096496af656fc607116"} Jul 28 19:36:46.347675 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.347601 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rqcnj" podStartSLOduration=5.604911931 podStartE2EDuration="24.347581496s" totalImagesPullingTime="18.742669565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.341158465 +0000 UTC m=+23.777112418" watchObservedRunningTime="2026-07-28 19:36:46.347581496 +0000 UTC m=+25.783535449" Jul 28 19:36:46.348012 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:46.347982 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pm7th" podStartSLOduration=3.337682445 podStartE2EDuration="24.347973138s" totalImagesPullingTime="21.010290693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:46.347256284 +0000 UTC m=+25.783210252" watchObservedRunningTime="2026-07-28 19:36:46.347973138 +0000 UTC m=+25.783927093" Jul 28 19:36:47.110878 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:47.110638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:47.111276 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:47.111256 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:47.156781 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:47.156746 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:47.156947 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:47.156890 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:47.335141 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:47.335068 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:47.335589 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:47.335465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82zr4" Jul 28 19:36:48.156299 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:48.156263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:48.156624 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:48.156309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:48.156624 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:48.156408 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:48.156624 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:48.156551 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:49.157111 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.156940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:49.157706 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:49.157197 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:49.341160 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.341124 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="a6f8f8410a79657d95274d8dc72980c31d39f6edb607ca12b430ce6f84737b18" exitCode=0 Jul 28 19:36:49.341324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.341198 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"a6f8f8410a79657d95274d8dc72980c31d39f6edb607ca12b430ce6f84737b18"} Jul 28 19:36:49.344654 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.344625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" event={"ID":"20acf618-f784-4e5a-b682-32ca663a056f","Type":"ContainerStarted","Data":"0d2ed85277060492517349f589b3b43b15d2fa44a0ab29cc8730d38ea340cd9c"} Jul 28 19:36:49.345320 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.344913 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:49.359726 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.359708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:49.387723 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:49.387657 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" podStartSLOduration=8.592923531 podStartE2EDuration="27.387645724s" totalImagesPullingTime="18.794722193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:49.386468901 +0000 UTC m=+28.822422843" watchObservedRunningTime="2026-07-28 19:36:49.387645724 +0000 UTC m=+28.823599675" Jul 28 19:36:50.156433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.156408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:50.156633 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.156408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:50.156633 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:50.156517 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:50.156633 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:50.156590 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:50.347324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.347293 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:50.347731 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.347365 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:50.368812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.368788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:36:50.807534 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.807446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kqqfs"] Jul 28 19:36:50.807921 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.807901 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:50.808050 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:50.808029 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:50.810004 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.809982 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46x8p"] Jul 28 19:36:50.810119 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.810067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:50.810196 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:50.810174 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:50.818788 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.818765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfcxs"] Jul 28 19:36:50.818904 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:50.818841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:50.818956 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:50.818913 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:51.351657 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:51.351623 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="92cd01f7d5ca93f1030c1bc2dd0848bee988ea680e21181936cf1e582d76b544" exitCode=0 Jul 28 19:36:51.352031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:51.351693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"92cd01f7d5ca93f1030c1bc2dd0848bee988ea680e21181936cf1e582d76b544"} Jul 28 19:36:52.156387 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:52.156184 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:52.156524 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:52.156210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:52.156524 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:52.156429 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:52.156524 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:52.156496 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:52.156524 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:52.156209 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:52.156680 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:52.156582 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:53.358354 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:53.358317 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="9b382fed72b4b224b4b49188312eea414b1a545bb0d64eec1ef10ee8b5d3441a" exitCode=0 Jul 28 19:36:53.358936 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:53.358369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"9b382fed72b4b224b4b49188312eea414b1a545bb0d64eec1ef10ee8b5d3441a"} Jul 28 19:36:54.156173 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.156147 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:54.156504 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.156463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:54.157494 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.156847 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfcxs" podUID="f8843a87-0c48-40f6-bc7b-4eed6db45745" Jul 28 19:36:54.157494 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.157003 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:54.159123 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.157822 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46x8p" podUID="d96ca0b6-60e2-4e21-af8c-83ee35e99958" Jul 28 19:36:54.159123 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.158373 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kqqfs" podUID="76938137-378f-4f48-99ab-dae7a5688d58" Jul 28 19:36:54.752021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.751991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:54.752021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.752035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752153 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752177 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752193 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752205 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4hnq4 for pod openshift-network-diagnostics/network-check-target-xfcxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752227 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs podName:d96ca0b6-60e2-4e21-af8c-83ee35e99958 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:26.752206594 +0000 UTC m=+66.188160525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs") pod "network-metrics-daemon-46x8p" (UID: "d96ca0b6-60e2-4e21-af8c-83ee35e99958") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:36:54.752632 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.752248 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4 podName:f8843a87-0c48-40f6-bc7b-4eed6db45745 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:26.75223717 +0000 UTC m=+66.188191100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hnq4" (UniqueName: "kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4") pod "network-check-target-xfcxs" (UID: "f8843a87-0c48-40f6-bc7b-4eed6db45745") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:36:54.852861 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:54.852830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:54.853043 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.852954 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:54.853043 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:54.853023 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret podName:76938137-378f-4f48-99ab-dae7a5688d58 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:26.853005505 +0000 UTC m=+66.288959445 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret") pod "global-pull-secret-syncer-kqqfs" (UID: "76938137-378f-4f48-99ab-dae7a5688d58") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:36:55.853443 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.853416 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-248.ec2.internal" event="NodeReady" Jul 28 19:36:55.853851 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.853556 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 19:36:55.890631 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.890589 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dvczk"] Jul 28 19:36:55.896031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.895999 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99"] Jul 28 19:36:55.896206 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.896173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:55.899652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.899628 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.899652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.899637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 19:36:55.900169 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.900007 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7bdcf959c-db97m"] Jul 28 19:36:55.900169 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.900120 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:55.900315 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.900282 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 19:36:55.901326 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.900747 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xk8bm\"" Jul 28 19:36:55.901326 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.901066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.902592 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.902565 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 19:36:55.903071 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.903774 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903753 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj"] Jul 28 19:36:55.903873 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903828 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.903930 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903893 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ghj27\"" Jul 28 19:36:55.903995 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903892 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:55.904323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.903986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 19:36:55.908129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.907969 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb"] Jul 28 19:36:55.908129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.907988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 19:36:55.908129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.908105 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:55.911368 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.911165 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 19:36:55.912146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 19:36:55.912146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912146 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.912289 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912164 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 19:36:55.912376 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912359 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z9wgr\"" Jul 28 19:36:55.912442 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912402 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-576sb"] Jul 28 19:36:55.912542 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" Jul 28 19:36:55.912597 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912580 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9s5j8\"" Jul 28 19:36:55.912851 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912698 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 19:36:55.912851 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.912785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 19:36:55.913073 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.913054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.913119 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.913096 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.913162 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.913117 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.916009 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.915942 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:36:55.916097 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.916054 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:55.918381 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.918361 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.919605 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.919588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:55.919991 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.919970 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.920110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920093 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 19:36:55.920182 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920137 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q4jl8\"" Jul 28 19:36:55.920555 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920533 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.920772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920750 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 19:36:55.920864 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920810 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.921308 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.920979 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dvczk"] Jul 28 19:36:55.922523 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.921655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jrmmr\"" Jul 28 19:36:55.922523 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.922472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj"] Jul 28 19:36:55.923590 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.923569 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb"] Jul 28 19:36:55.925116 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.925093 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:36:55.925994 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.925977 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh"] Jul 28 19:36:55.926129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.926111 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 19:36:55.926210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.926048 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 19:36:55.926303 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.926282 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 19:36:55.926489 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.926469 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t265\"" Jul 28 19:36:55.930234 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.930216 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g"] Jul 28 19:36:55.930348 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.930329 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" Jul 28 19:36:55.932239 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.932219 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 19:36:55.934113 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.933967 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p"] Jul 28 19:36:55.934113 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.934051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.934252 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.934123 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 19:36:55.934252 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.934158 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.934423 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.934397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:55.936225 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.935594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mpv5p\"" Jul 28 19:36:55.937804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.937696 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bdcf959c-db97m"] Jul 28 19:36:55.937804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.937730 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lk7lh"] Jul 28 19:36:55.937917 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.937808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:55.938797 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.938770 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 19:36:55.938887 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.938795 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.939342 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.939325 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 19:36:55.939443 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.939408 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d6s56\"" Jul 28 19:36:55.939672 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.939637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.940611 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.940572 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 19:36:55.941489 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.940887 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lvvgp\"" Jul 28 19:36:55.941489 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.941012 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-576sb"] Jul 28 19:36:55.941489 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.941132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:55.941984 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.941938 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 19:36:55.942897 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.942865 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.943236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.943214 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g"] Jul 28 19:36:55.943857 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.943840 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.944521 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.944504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 19:36:55.944899 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.944554 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 19:36:55.944899 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.944512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pkrrg\"" Jul 28 19:36:55.945366 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.945323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99"] Jul 28 19:36:55.948151 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.948131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lk7lh"] Jul 28 19:36:55.952464 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.952442 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh"] Jul 28 19:36:55.959960 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.959942 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-65trs"] Jul 28 19:36:55.963396 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.963374 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p"] Jul 28 19:36:55.963479 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.963468 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:55.966101 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.966085 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 19:36:55.966227 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.966099 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-phccc\"" Jul 28 19:36:55.966227 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.966224 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 19:36:55.967952 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.966294 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 19:36:55.972984 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:55.972959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65trs"] Jul 28 19:36:56.059916 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.059886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.059916 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.059917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-tmp\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.060117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.059935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.059953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnj2b\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr4j\" (UniqueName: \"kubernetes.io/projected/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-kube-api-access-dtr4j\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-stats-auth\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956n9\" (UniqueName: \"kubernetes.io/projected/61168e6a-d37c-46a1-9b88-2e0e635c81b9-kube-api-access-956n9\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58snk\" (UniqueName: \"kubernetes.io/projected/89664ed0-d7e1-4548-bf7c-d923c48b3ede-kube-api-access-58snk\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-config-volume\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.060360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7k5j\" (UniqueName: \"kubernetes.io/projected/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-kube-api-access-q7k5j\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9c5\" (UniqueName: \"kubernetes.io/projected/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-kube-api-access-sv9c5\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61168e6a-d37c-46a1-9b88-2e0e635c81b9-cert\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89664ed0-d7e1-4548-bf7c-d923c48b3ede-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-config\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060587 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tc2x\" (UniqueName: \"kubernetes.io/projected/9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2-kube-api-access-9tc2x\") pod \"volume-data-source-validator-549c6d54cf-hrvzb\" (UID: \"9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvch\" (UniqueName: \"kubernetes.io/projected/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-kube-api-access-vhvch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92eb4695-0595-47ed-8a51-ad1eff1364eb-config\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.060705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-service-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-trusted-ca\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/dce9f717-6cac-46ae-9efc-70444d11deea-kube-api-access-q6hv8\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-snapshots\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-serving-cert\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.060977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dce9f717-6cac-46ae-9efc-70444d11deea-serving-cert\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92eb4695-0595-47ed-8a51-ad1eff1364eb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wjh\" (UniqueName: \"kubernetes.io/projected/92eb4695-0595-47ed-8a51-ad1eff1364eb-kube-api-access-w4wjh\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzbj\" (UniqueName: \"kubernetes.io/projected/4138a42b-aed5-43c5-9b0c-e67f9467e8b2-kube-api-access-xpzbj\") pod \"network-check-source-7d88cb55d8-9jmzh\" (UID: \"4138a42b-aed5-43c5-9b0c-e67f9467e8b2\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-tmp-dir\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.061218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-default-certificate\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.061684 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.061684 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.061256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4jf\" (UniqueName: \"kubernetes.io/projected/630164a9-08ce-432d-ba5a-f2047a822a82-kube-api-access-kn4jf\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.156743 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.156719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:36:56.156892 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.156823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:36:56.156959 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.156934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:36:56.159377 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.159301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 19:36:56.159480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.159463 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6gvt\"" Jul 28 19:36:56.159602 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.159572 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfqx7\"" Jul 28 19:36:56.159898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.159881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 19:36:56.161565 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.161694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.161694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr4j\" (UniqueName: \"kubernetes.io/projected/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-kube-api-access-dtr4j\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.161694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-stats-auth\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.161694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956n9\" (UniqueName: \"kubernetes.io/projected/61168e6a-d37c-46a1-9b88-2e0e635c81b9-kube-api-access-956n9\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58snk\" (UniqueName: \"kubernetes.io/projected/89664ed0-d7e1-4548-bf7c-d923c48b3ede-kube-api-access-58snk\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-config-volume\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7k5j\" (UniqueName: \"kubernetes.io/projected/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-kube-api-access-q7k5j\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9c5\" (UniqueName: \"kubernetes.io/projected/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-kube-api-access-sv9c5\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.161914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61168e6a-d37c-46a1-9b88-2e0e635c81b9-cert\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.161978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89664ed0-d7e1-4548-bf7c-d923c48b3ede-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-config\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tc2x\" (UniqueName: \"kubernetes.io/projected/9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2-kube-api-access-9tc2x\") pod \"volume-data-source-validator-549c6d54cf-hrvzb\" (UID: \"9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162078 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvch\" (UniqueName: \"kubernetes.io/projected/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-kube-api-access-vhvch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92eb4695-0595-47ed-8a51-ad1eff1364eb-config\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-service-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.162288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-trusted-ca\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.162294 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.162357 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.662339328 +0000 UTC m=+36.098293261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/dce9f717-6cac-46ae-9efc-70444d11deea-kube-api-access-q6hv8\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-snapshots\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-serving-cert\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.162823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dce9f717-6cac-46ae-9efc-70444d11deea-serving-cert\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92eb4695-0595-47ed-8a51-ad1eff1364eb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-config-volume\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.162997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-service-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.163072 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.163119 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.66310317 +0000 UTC m=+36.099057101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.163123 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.163140 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:36:56.163210 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.163171 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.663159997 +0000 UTC m=+36.099113928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:36:56.163688 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.163279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.163688 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.163535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-snapshots\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.163688 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.163564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.163977 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.163952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92eb4695-0595-47ed-8a51-ad1eff1364eb-config\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.164117 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.664099467 +0000 UTC m=+36.100053416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wjh\" (UniqueName: \"kubernetes.io/projected/92eb4695-0595-47ed-8a51-ad1eff1364eb-kube-api-access-w4wjh\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzbj\" (UniqueName: \"kubernetes.io/projected/4138a42b-aed5-43c5-9b0c-e67f9467e8b2-kube-api-access-xpzbj\") pod \"network-check-source-7d88cb55d8-9jmzh\" (UID: \"4138a42b-aed5-43c5-9b0c-e67f9467e8b2\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-tmp-dir\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164338 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-default-certificate\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4jf\" (UniqueName: \"kubernetes.io/projected/630164a9-08ce-432d-ba5a-f2047a822a82-kube-api-access-kn4jf\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-tmp\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-config\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.164898 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnj2b\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.164807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dce9f717-6cac-46ae-9efc-70444d11deea-trusted-ca\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.164823 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.164868 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.664854532 +0000 UTC m=+36.100808464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.165157 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.165223 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.665205873 +0000 UTC m=+36.101159806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:36:56.165746 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.165653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-tmp-dir\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.166042 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.165913 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-tmp\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.167010 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.166553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/89664ed0-d7e1-4548-bf7c-d923c48b3ede-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.167010 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.166641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.167789 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.167762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61168e6a-d37c-46a1-9b88-2e0e635c81b9-cert\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.167864 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.167765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dce9f717-6cac-46ae-9efc-70444d11deea-serving-cert\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.167966 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.167944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92eb4695-0595-47ed-8a51-ad1eff1364eb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.169844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.170341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-default-certificate\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.170605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-stats-auth\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.171310 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.171920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.174345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4jf\" (UniqueName: \"kubernetes.io/projected/630164a9-08ce-432d-ba5a-f2047a822a82-kube-api-access-kn4jf\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.174447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr4j\" (UniqueName: \"kubernetes.io/projected/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-kube-api-access-dtr4j\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.175055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9c5\" (UniqueName: \"kubernetes.io/projected/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-kube-api-access-sv9c5\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.175063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58snk\" (UniqueName: \"kubernetes.io/projected/89664ed0-d7e1-4548-bf7c-d923c48b3ede-kube-api-access-58snk\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.175346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.176149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.175606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tc2x\" (UniqueName: \"kubernetes.io/projected/9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2-kube-api-access-9tc2x\") pod \"volume-data-source-validator-549c6d54cf-hrvzb\" (UID: \"9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" Jul 28 19:36:56.176747 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.176198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c57e53-4a0a-4b42-ade0-44ed7a500ed7-serving-cert\") pod \"insights-operator-5885fdb759-dvczk\" (UID: \"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7\") " pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.177745 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.177375 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvch\" (UniqueName: \"kubernetes.io/projected/45bcbaaa-d4fd-4c39-89fb-3c05b8102493-kube-api-access-vhvch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g\" (UID: \"45bcbaaa-d4fd-4c39-89fb-3c05b8102493\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.177745 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.177563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/dce9f717-6cac-46ae-9efc-70444d11deea-kube-api-access-q6hv8\") pod \"console-operator-584dcb6b54-576sb\" (UID: \"dce9f717-6cac-46ae-9efc-70444d11deea\") " pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.177745 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.177702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7k5j\" (UniqueName: \"kubernetes.io/projected/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-kube-api-access-q7k5j\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.178065 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.178043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnj2b\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.178535 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.178518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-956n9\" (UniqueName: \"kubernetes.io/projected/61168e6a-d37c-46a1-9b88-2e0e635c81b9-kube-api-access-956n9\") pod \"ingress-canary-65trs\" (UID: \"61168e6a-d37c-46a1-9b88-2e0e635c81b9\") " pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.178773 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.178741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzbj\" (UniqueName: \"kubernetes.io/projected/4138a42b-aed5-43c5-9b0c-e67f9467e8b2-kube-api-access-xpzbj\") pod \"network-check-source-7d88cb55d8-9jmzh\" (UID: \"4138a42b-aed5-43c5-9b0c-e67f9467e8b2\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" Jul 28 19:36:56.179230 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.179208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wjh\" (UniqueName: \"kubernetes.io/projected/92eb4695-0595-47ed-8a51-ad1eff1364eb-kube-api-access-w4wjh\") pod \"service-ca-operator-5d6c75f84f-zbr5p\" (UID: \"92eb4695-0595-47ed-8a51-ad1eff1364eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.212893 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.212867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dvczk" Jul 28 19:36:56.251339 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.251308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" Jul 28 19:36:56.260589 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.260565 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:36:56.275876 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.275849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" Jul 28 19:36:56.282891 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.282868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" Jul 28 19:36:56.290727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.290707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" Jul 28 19:36:56.303955 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.303932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65trs" Jul 28 19:36:56.671125 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.671313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:56.671313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:56.671313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:56.671313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:56.671313 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671297 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671271127 +0000 UTC m=+37.107225080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671352 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671354 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671374 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:56.671363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671394 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671414 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671399 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671385241 +0000 UTC m=+37.107339204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671353 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671463 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671450741 +0000 UTC m=+37.107404690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671483 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671473139 +0000 UTC m=+37.107427081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671502 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671496306 +0000 UTC m=+37.107450236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:36:56.671583 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:56.671522 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.671515838 +0000 UTC m=+37.107469767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:36:57.528848 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.528493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p"] Jul 28 19:36:57.547152 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.547127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dvczk"] Jul 28 19:36:57.551028 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:57.550992 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c57e53_4a0a_4b42_ade0_44ed7a500ed7.slice/crio-c20feae6d6c742389b60dac864fe164c180a8f3fbe1127cfdcf614276da3fee9 WatchSource:0}: Error finding container c20feae6d6c742389b60dac864fe164c180a8f3fbe1127cfdcf614276da3fee9: Status 404 returned error can't find the container with id c20feae6d6c742389b60dac864fe164c180a8f3fbe1127cfdcf614276da3fee9 Jul 28 19:36:57.551144 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.551036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb"] Jul 28 19:36:57.555141 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.554830 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g"] Jul 28 19:36:57.556527 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.556440 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-576sb"] Jul 28 19:36:57.556600 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:57.556544 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2e98f5_0401_4e17_b78a_2a3fdd5eeed2.slice/crio-f4aceb70215df48ca571dc61e58e8a4320b39d7a2da14209fb194ff09eddfaec WatchSource:0}: Error finding container f4aceb70215df48ca571dc61e58e8a4320b39d7a2da14209fb194ff09eddfaec: Status 404 returned error can't find the container with id f4aceb70215df48ca571dc61e58e8a4320b39d7a2da14209fb194ff09eddfaec Jul 28 19:36:57.558224 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.558152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65trs"] Jul 28 19:36:57.571951 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.571933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh"] Jul 28 19:36:57.574773 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:57.574752 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61168e6a_d37c_46a1_9b88_2e0e635c81b9.slice/crio-78b106de20ba648d0f81c164d405e4dbc127a018a5d410ca568665628cf5e034 WatchSource:0}: Error finding container 78b106de20ba648d0f81c164d405e4dbc127a018a5d410ca568665628cf5e034: Status 404 returned error can't find the container with id 78b106de20ba648d0f81c164d405e4dbc127a018a5d410ca568665628cf5e034 Jul 28 19:36:57.576034 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:36:57.576015 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4138a42b_aed5_43c5_9b0c_e67f9467e8b2.slice/crio-6dbb464e0b5f379033614c8bd52d84bb11397ab8d8dc3f36ad5dea5331fdc559 WatchSource:0}: Error finding container 6dbb464e0b5f379033614c8bd52d84bb11397ab8d8dc3f36ad5dea5331fdc559: Status 404 returned error can't find the container with id 6dbb464e0b5f379033614c8bd52d84bb11397ab8d8dc3f36ad5dea5331fdc559 Jul 28 19:36:57.689220 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:57.689321 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:57.689321 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:57.689412 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689347 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:36:57.689412 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689367 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:36:57.689412 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689367 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:36:57.689412 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689410 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.689406682 +0000 UTC m=+39.125360619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689451 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.689444756 +0000 UTC m=+39.125398685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689469 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.68946419 +0000 UTC m=+39.125418119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689486 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.689477526 +0000 UTC m=+39.125431462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:57.689539 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:57.689805 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:57.689561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:57.689805 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689631 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:36:57.689805 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689644 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:36:57.689805 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689687 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.689654892 +0000 UTC m=+39.125608828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:36:57.689805 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:57.689700 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.689694831 +0000 UTC m=+39.125648761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:36:58.390597 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.390514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" event={"ID":"4138a42b-aed5-43c5-9b0c-e67f9467e8b2","Type":"ContainerStarted","Data":"6dbb464e0b5f379033614c8bd52d84bb11397ab8d8dc3f36ad5dea5331fdc559"} Jul 28 19:36:58.395075 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.394674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" event={"ID":"92eb4695-0595-47ed-8a51-ad1eff1364eb","Type":"ContainerStarted","Data":"6cf302b2d39b5a0fca839957e10cb2c7d0f93ab5a7d6dda83eb9674137c660cc"} Jul 28 19:36:58.400372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.399598 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="32397c51fc98d515d2a3a30274988a45a6bf93cb92aea95dc2d319498cd642a9" exitCode=0 Jul 28 19:36:58.400372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.399699 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"32397c51fc98d515d2a3a30274988a45a6bf93cb92aea95dc2d319498cd642a9"} Jul 28 19:36:58.403730 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.402803 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" event={"ID":"45bcbaaa-d4fd-4c39-89fb-3c05b8102493","Type":"ContainerStarted","Data":"72cbe909cedf3e6bcfc820e8bc00e7ad0ab5800e590b8a2899dbbb6cc81e5d30"} Jul 28 19:36:58.406931 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.406872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65trs" event={"ID":"61168e6a-d37c-46a1-9b88-2e0e635c81b9","Type":"ContainerStarted","Data":"78b106de20ba648d0f81c164d405e4dbc127a018a5d410ca568665628cf5e034"} Jul 28 19:36:58.411480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.411418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dvczk" event={"ID":"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7","Type":"ContainerStarted","Data":"c20feae6d6c742389b60dac864fe164c180a8f3fbe1127cfdcf614276da3fee9"} Jul 28 19:36:58.413938 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.413873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" event={"ID":"9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2","Type":"ContainerStarted","Data":"f4aceb70215df48ca571dc61e58e8a4320b39d7a2da14209fb194ff09eddfaec"} Jul 28 19:36:58.416134 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:58.416079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" event={"ID":"dce9f717-6cac-46ae-9efc-70444d11deea","Type":"ContainerStarted","Data":"54e54d793911a233fcdd7bf8d0a70dbba0d7d32ef3d89bbb96c3d4e5788434be"} Jul 28 19:36:59.428959 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.428023 2562 generic.go:356] "Generic (PLEG): container finished" podID="76415a4a-5a70-4b6d-bf8b-2a38258a58ce" containerID="fc10f349a3e5e3ec542d56bbf7767af5c3a5c5b83eb30a66d6eba2962b511c30" exitCode=0 Jul 28 19:36:59.428959 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.428097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerDied","Data":"fc10f349a3e5e3ec542d56bbf7767af5c3a5c5b83eb30a66d6eba2962b511c30"} Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.711853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.711943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.711980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.712017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.712042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:36:59.712086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712227 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712286 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.712268057 +0000 UTC m=+43.148221994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712719 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.712702161 +0000 UTC m=+43.148656094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712795 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712827 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.712816292 +0000 UTC m=+43.148770225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712882 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712912 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.712901788 +0000 UTC m=+43.148855720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712972 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:36:59.713244 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.712982 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:36:59.714130 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.713026 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.713015961 +0000 UTC m=+43.148969904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:36:59.714130 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.713093 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:36:59.714130 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:36:59.713124 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.71311499 +0000 UTC m=+43.149068925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:37:00.436562 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:00.436525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nlv45" event={"ID":"76415a4a-5a70-4b6d-bf8b-2a38258a58ce","Type":"ContainerStarted","Data":"69c2985da4ad52f5f8e4727a04b7e4bb1330b122f555f036beb9d431248cfed9"} Jul 28 19:37:00.438428 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:00.438383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" event={"ID":"9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2","Type":"ContainerStarted","Data":"a7d5ffd09ed847b37f89b546848ffeab9ee073cac22047c89884770a11d4a83a"} Jul 28 19:37:00.460996 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:00.460953 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nlv45" podStartSLOduration=8.360544795 podStartE2EDuration="38.460937891s" totalImagesPullingTime="30.100393096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:00.459193071 +0000 UTC m=+39.895147029" watchObservedRunningTime="2026-07-28 19:37:00.460937891 +0000 UTC m=+39.896891821" Jul 28 19:37:00.474959 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:00.474874 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-hrvzb" podStartSLOduration=29.148851517 podStartE2EDuration="31.474856041s" totalImagesPullingTime="2.326004524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:00.474601126 +0000 UTC m=+39.910555077" watchObservedRunningTime="2026-07-28 19:37:00.474856041 +0000 UTC m=+39.910809994" Jul 28 19:37:03.750510 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:03.750642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750676 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.750639403 +0000 UTC m=+51.186593356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750713 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750752 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750756 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750782 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.750763875 +0000 UTC m=+51.186717827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750783 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750823 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.750806503 +0000 UTC m=+51.186760432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750830 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750766 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750842 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.750832869 +0000 UTC m=+51.186786801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:37:03.751013 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750938 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.750909479 +0000 UTC m=+51.186863422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:37:03.751607 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:03.750963 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.75095294 +0000 UTC m=+51.186906875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:37:05.455240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:05.455212 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dvczk" event={"ID":"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7","Type":"ContainerStarted","Data":"f55bd36daef3cbf814d7c36a310db52a12c4c55b64daf3ec0b9f668a648e2e93"} Jul 28 19:37:05.457056 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:05.457030 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" event={"ID":"92eb4695-0595-47ed-8a51-ad1eff1364eb","Type":"ContainerStarted","Data":"76c138cb4cc927be6cbfbd745b731a48604d25a31ec563f7858365215fb807d6"} Jul 28 19:37:05.485274 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:05.485093 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-dvczk" podStartSLOduration=31.728878219 podStartE2EDuration="39.48507343s" totalImagesPullingTime="7.756195211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:05.483520349 +0000 UTC m=+44.919474314" watchObservedRunningTime="2026-07-28 19:37:05.48507343 +0000 UTC m=+44.921027384" Jul 28 19:37:06.463016 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.462984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/0.log" Jul 28 19:37:06.463442 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.463033 2562 generic.go:356] "Generic (PLEG): container finished" podID="dce9f717-6cac-46ae-9efc-70444d11deea" containerID="0032f4f6b8b7b7ef0fd96b65df25f900033576c8e9da6fd8da33fc8a6e11ed5e" exitCode=255 Jul 28 19:37:06.463442 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.463070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" event={"ID":"dce9f717-6cac-46ae-9efc-70444d11deea","Type":"ContainerDied","Data":"0032f4f6b8b7b7ef0fd96b65df25f900033576c8e9da6fd8da33fc8a6e11ed5e"} Jul 28 19:37:06.463442 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.463348 2562 scope.go:117] "RemoveContainer" containerID="0032f4f6b8b7b7ef0fd96b65df25f900033576c8e9da6fd8da33fc8a6e11ed5e" Jul 28 19:37:06.465562 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.465533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" event={"ID":"4138a42b-aed5-43c5-9b0c-e67f9467e8b2","Type":"ContainerStarted","Data":"a89d20f7256b753bd0cbd32d3091c79a02212b7be1a2b97013ee5e4f3731eba1"} Jul 28 19:37:06.467237 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.467216 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" event={"ID":"45bcbaaa-d4fd-4c39-89fb-3c05b8102493","Type":"ContainerStarted","Data":"724aabafab3577a3c3651ba9dcb7cfed3882fcbf1c0f3fee189dd56da2d774ca"} Jul 28 19:37:06.468788 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.468767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65trs" event={"ID":"61168e6a-d37c-46a1-9b88-2e0e635c81b9","Type":"ContainerStarted","Data":"b90305cdf16d0ae46ccd295198bec45dcfea875a9f2eea3314656cae598d5034"} Jul 28 19:37:06.494685 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.494612 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jmzh" podStartSLOduration=23.76112555 podStartE2EDuration="31.494594225s" totalImagesPullingTime="7.733468675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:06.493140665 +0000 UTC m=+45.929094618" watchObservedRunningTime="2026-07-28 19:37:06.494594225 +0000 UTC m=+45.930548178" Jul 28 19:37:06.538981 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.538844 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-65trs" podStartSLOduration=3.796960271 podStartE2EDuration="11.538822567s" totalImagesPullingTime="7.741862296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:06.538118472 +0000 UTC m=+45.974072424" watchObservedRunningTime="2026-07-28 19:37:06.538822567 +0000 UTC m=+45.974776522" Jul 28 19:37:06.539208 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.539181 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" podStartSLOduration=22.77026067 podStartE2EDuration="30.53917242s" totalImagesPullingTime="7.76891175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:06.517218504 +0000 UTC m=+45.953172456" watchObservedRunningTime="2026-07-28 19:37:06.53917242 +0000 UTC m=+45.975126396" Jul 28 19:37:06.554784 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.554636 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" podStartSLOduration=23.80683643 podStartE2EDuration="31.554620157s" totalImagesPullingTime="7.747783727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:06.55414105 +0000 UTC m=+45.990095006" watchObservedRunningTime="2026-07-28 19:37:06.554620157 +0000 UTC m=+45.990574111" Jul 28 19:37:06.739371 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.738867 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkv6s"] Jul 28 19:37:06.752170 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.752143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.754581 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.754562 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdlcv\"" Jul 28 19:37:06.755021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.754999 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 19:37:06.755395 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.755378 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 19:37:06.759654 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.759625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkv6s"] Jul 28 19:37:06.779180 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.779155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbcc\" (UniqueName: \"kubernetes.io/projected/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-api-access-kkbcc\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.779312 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.779242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-crio-socket\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.779312 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.779264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.779312 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.779285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-data-volume\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.779434 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.779311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880502 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-crio-socket\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880502 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880502 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-data-volume\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-crio-socket\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:06.880559 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbcc\" (UniqueName: \"kubernetes.io/projected/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-api-access-kkbcc\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:06.880730 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:07.380702302 +0000 UTC m=+46.816656246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : configmap "kube-rbac-proxy" not found Jul 28 19:37:06.880792 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:06.880787 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 19:37:06.881027 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.880837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-data-volume\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:06.881027 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:06.880863 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:07.380845574 +0000 UTC m=+46.816799517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : secret "insights-runtime-extractor-tls" not found Jul 28 19:37:06.892057 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:06.892034 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbcc\" (UniqueName: \"kubernetes.io/projected/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-api-access-kkbcc\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:07.384336 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.384308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:07.384528 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.384372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:07.384528 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:07.384427 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 19:37:07.384528 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:07.384487 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:08.384472784 +0000 UTC m=+47.820426714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : configmap "kube-rbac-proxy" not found Jul 28 19:37:07.384696 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:07.384525 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 19:37:07.384696 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:07.384577 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:08.384564112 +0000 UTC m=+47.820518042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : secret "insights-runtime-extractor-tls" not found Jul 28 19:37:07.473127 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473101 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:37:07.473511 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/0.log" Jul 28 19:37:07.473551 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473524 2562 generic.go:356] "Generic (PLEG): container finished" podID="dce9f717-6cac-46ae-9efc-70444d11deea" containerID="cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75" exitCode=255 Jul 28 19:37:07.473623 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" event={"ID":"dce9f717-6cac-46ae-9efc-70444d11deea","Type":"ContainerDied","Data":"cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75"} Jul 28 19:37:07.473700 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473686 2562 scope.go:117] "RemoveContainer" containerID="0032f4f6b8b7b7ef0fd96b65df25f900033576c8e9da6fd8da33fc8a6e11ed5e" Jul 28 19:37:07.473914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:07.473892 2562 scope.go:117] "RemoveContainer" containerID="cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75" Jul 28 19:37:07.474115 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:07.474096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-576sb_openshift-console-operator(dce9f717-6cac-46ae-9efc-70444d11deea)\"" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" podUID="dce9f717-6cac-46ae-9efc-70444d11deea" Jul 28 19:37:08.335449 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:08.335420 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flg66_023e36af-fadf-47f4-93de-719edd61b816/dns-node-resolver/0.log" Jul 28 19:37:08.394891 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:08.394861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:08.395029 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:08.394908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:08.395029 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:08.394993 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 19:37:08.395096 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:08.395037 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 19:37:08.395096 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:08.395055 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:10.395040738 +0000 UTC m=+49.830994672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : configmap "kube-rbac-proxy" not found Jul 28 19:37:08.395096 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:08.395076 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:10.395063505 +0000 UTC m=+49.831017435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : secret "insights-runtime-extractor-tls" not found Jul 28 19:37:08.478039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:08.478013 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:37:08.478392 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:08.478369 2562 scope.go:117] "RemoveContainer" containerID="cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75" Jul 28 19:37:08.478543 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:08.478524 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-576sb_openshift-console-operator(dce9f717-6cac-46ae-9efc-70444d11deea)\"" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" podUID="dce9f717-6cac-46ae-9efc-70444d11deea" Jul 28 19:37:09.136869 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:09.136841 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6g2s4_8e7a676f-2b30-4b7a-ba6b-606a21ca6068/node-ca/0.log" Jul 28 19:37:09.937210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:09.937180 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-65trs_61168e6a-d37c-46a1-9b88-2e0e635c81b9/serve-healthcheck-canary/0.log" Jul 28 19:37:10.411268 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:10.411240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:10.411409 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:10.411288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:10.411409 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:10.411356 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 19:37:10.411409 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:10.411371 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 19:37:10.411506 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:10.411436 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:14.411418673 +0000 UTC m=+53.847372604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : secret "insights-runtime-extractor-tls" not found Jul 28 19:37:10.411506 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:10.411454 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:14.411445893 +0000 UTC m=+53.847399824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : configmap "kube-rbac-proxy" not found Jul 28 19:37:11.823519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823640 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:11.823711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823733 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls podName:ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823715728 +0000 UTC m=+67.259669659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls") pod "dns-default-lk7lh" (UID: "ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f") : secret "dns-default-metrics-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823737 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823760 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823733 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823784 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cbbfd6c4c-csfq7: secret "image-registry-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823787 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823776808 +0000 UTC m=+67.259730738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : configmap references non-existent config key: service-ca.crt Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823809 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs podName:630164a9-08ce-432d-ba5a-f2047a822a82 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823797798 +0000 UTC m=+67.259751736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs") pod "router-default-7bdcf959c-db97m" (UID: "630164a9-08ce-432d-ba5a-f2047a822a82") : secret "router-metrics-certs-default" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823826 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls podName:d101e2cd-cac4-4e7a-ba14-1a4f604f7de1 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823817998 +0000 UTC m=+67.259771928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls") pod "cluster-samples-operator-6748489656-n4srj" (UID: "d101e2cd-cac4-4e7a-ba14-1a4f604f7de1") : secret "samples-operator-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823829 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 19:37:11.823953 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823841 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls podName:092fbc21-4f74-489b-9941-edf6694f4c23 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823833119 +0000 UTC m=+67.259787050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls") pod "image-registry-7cbbfd6c4c-csfq7" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23") : secret "image-registry-tls" not found Jul 28 19:37:11.824512 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:11.823877 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls podName:89664ed0-d7e1-4548-bf7c-d923c48b3ede nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.823868918 +0000 UTC m=+67.259822851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tbm99" (UID: "89664ed0-d7e1-4548-bf7c-d923c48b3ede") : secret "cluster-monitoring-operator-tls" not found Jul 28 19:37:14.447244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:14.447212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:14.447623 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:14.447264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:14.447623 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:14.447351 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 19:37:14.447623 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:14.447401 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls podName:86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:22.447388485 +0000 UTC m=+61.883342416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkv6s" (UID: "86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5") : secret "insights-runtime-extractor-tls" not found Jul 28 19:37:14.447791 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:14.447768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:16.261473 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:16.261438 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:37:16.262140 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:16.261503 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:37:16.262140 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:16.261932 2562 scope.go:117] "RemoveContainer" containerID="cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75" Jul 28 19:37:16.262261 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:16.262138 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-576sb_openshift-console-operator(dce9f717-6cac-46ae-9efc-70444d11deea)\"" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" podUID="dce9f717-6cac-46ae-9efc-70444d11deea" Jul 28 19:37:22.377920 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.377890 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ph2m" Jul 28 19:37:22.509714 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.509677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:22.511908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.511864 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkv6s\" (UID: \"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5\") " pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:22.666436 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.666402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdlcv\"" Jul 28 19:37:22.674359 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.674334 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkv6s" Jul 28 19:37:22.794456 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:22.794428 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkv6s"] Jul 28 19:37:22.797434 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:22.797409 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86af7b69_99fe_4c8d_8c4b_3ed9cd4420f5.slice/crio-6faf96eab8ceb4faec2d77b3a617513a8386b135afaa29b77e42dbcbd107909b WatchSource:0}: Error finding container 6faf96eab8ceb4faec2d77b3a617513a8386b135afaa29b77e42dbcbd107909b: Status 404 returned error can't find the container with id 6faf96eab8ceb4faec2d77b3a617513a8386b135afaa29b77e42dbcbd107909b Jul 28 19:37:23.526077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:23.525989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkv6s" event={"ID":"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5","Type":"ContainerStarted","Data":"6faf96eab8ceb4faec2d77b3a617513a8386b135afaa29b77e42dbcbd107909b"} Jul 28 19:37:26.850630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.850597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:37:26.851093 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.850678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:37:26.853211 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.853187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnq4\" (UniqueName: \"kubernetes.io/projected/f8843a87-0c48-40f6-bc7b-4eed6db45745-kube-api-access-4hnq4\") pod \"network-check-target-xfcxs\" (UID: \"f8843a87-0c48-40f6-bc7b-4eed6db45745\") " pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:37:26.853656 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.853616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 19:37:26.863741 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.863719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d96ca0b6-60e2-4e21-af8c-83ee35e99958-metrics-certs\") pod \"network-metrics-daemon-46x8p\" (UID: \"d96ca0b6-60e2-4e21-af8c-83ee35e99958\") " pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:37:26.951788 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.951748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:37:26.954833 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.954807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 19:37:26.964986 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:26.964962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76938137-378f-4f48-99ab-dae7a5688d58-original-pull-secret\") pod \"global-pull-secret-syncer-kqqfs\" (UID: \"76938137-378f-4f48-99ab-dae7a5688d58\") " pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:37:27.103541 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.103464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kqqfs" Jul 28 19:37:27.113175 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.113152 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfqx7\"" Jul 28 19:37:27.119456 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.119432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6gvt\"" Jul 28 19:37:27.121473 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.121454 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:37:27.127997 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.127979 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46x8p" Jul 28 19:37:27.655566 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.655503 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46x8p"] Jul 28 19:37:27.663611 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:27.663573 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96ca0b6_60e2_4e21_af8c_83ee35e99958.slice/crio-292a616bddb8356051d28461a3e36804e7983ad23bb51c6f40c3c6b92e5c51ed WatchSource:0}: Error finding container 292a616bddb8356051d28461a3e36804e7983ad23bb51c6f40c3c6b92e5c51ed: Status 404 returned error can't find the container with id 292a616bddb8356051d28461a3e36804e7983ad23bb51c6f40c3c6b92e5c51ed Jul 28 19:37:27.672916 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.672889 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfcxs"] Jul 28 19:37:27.674688 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:27.674634 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8843a87_0c48_40f6_bc7b_4eed6db45745.slice/crio-54f696facc0b840a53625c32ad74345d7d258d084a7092a515af4268b3745e2d WatchSource:0}: Error finding container 54f696facc0b840a53625c32ad74345d7d258d084a7092a515af4268b3745e2d: Status 404 returned error can't find the container with id 54f696facc0b840a53625c32ad74345d7d258d084a7092a515af4268b3745e2d Jul 28 19:37:27.690801 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.690780 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kqqfs"] Jul 28 19:37:27.694192 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:27.694170 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76938137_378f_4f48_99ab_dae7a5688d58.slice/crio-b9ecd31b76da874db04a7e6080586432b9be1d56aaf83ce580d812fe7b094df2 WatchSource:0}: Error finding container b9ecd31b76da874db04a7e6080586432b9be1d56aaf83ce580d812fe7b094df2: Status 404 returned error can't find the container with id b9ecd31b76da874db04a7e6080586432b9be1d56aaf83ce580d812fe7b094df2 Jul 28 19:37:27.859200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:27.859200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:37:27.859652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:27.859652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:27.859652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:37:27.859652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.859409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:27.860211 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.860174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630164a9-08ce-432d-ba5a-f2047a822a82-service-ca-bundle\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:27.862100 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.862073 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"image-registry-7cbbfd6c4c-csfq7\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:27.862211 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.862143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630164a9-08ce-432d-ba5a-f2047a822a82-metrics-certs\") pod \"router-default-7bdcf959c-db97m\" (UID: \"630164a9-08ce-432d-ba5a-f2047a822a82\") " pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:27.862211 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.862147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d101e2cd-cac4-4e7a-ba14-1a4f604f7de1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-n4srj\" (UID: \"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:37:27.862211 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.862146 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/89664ed0-d7e1-4548-bf7c-d923c48b3ede-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tbm99\" (UID: \"89664ed0-d7e1-4548-bf7c-d923c48b3ede\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:37:27.862488 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:27.862471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f-metrics-tls\") pod \"dns-default-lk7lh\" (UID: \"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f\") " pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:28.023978 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.023951 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ghj27\"" Jul 28 19:37:28.032852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.032820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" Jul 28 19:37:28.035096 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.035069 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9s5j8\"" Jul 28 19:37:28.043122 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.043101 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:28.045150 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.045119 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z9wgr\"" Jul 28 19:37:28.053945 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.053884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" Jul 28 19:37:28.070720 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.070697 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t265\"" Jul 28 19:37:28.079384 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.079320 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:28.103367 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.103146 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pkrrg\"" Jul 28 19:37:28.112518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.112432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:28.158800 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.158551 2562 scope.go:117] "RemoveContainer" containerID="cbed932d3224ada8b6a0a89039ae3e6ead555a2ed35d553041b64a56bc5bda75" Jul 28 19:37:28.214538 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.214447 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99"] Jul 28 19:37:28.265365 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.265280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bdcf959c-db97m"] Jul 28 19:37:28.272130 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:28.272101 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630164a9_08ce_432d_ba5a_f2047a822a82.slice/crio-4f25edf853b1c90192a8c0a25013c961a61cf7ddc864a35d483d16d49dd9d361 WatchSource:0}: Error finding container 4f25edf853b1c90192a8c0a25013c961a61cf7ddc864a35d483d16d49dd9d361: Status 404 returned error can't find the container with id 4f25edf853b1c90192a8c0a25013c961a61cf7ddc864a35d483d16d49dd9d361 Jul 28 19:37:28.285222 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.285149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj"] Jul 28 19:37:28.311483 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.311408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:37:28.319150 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:28.319077 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092fbc21_4f74_489b_9941_edf6694f4c23.slice/crio-32e7654acc7f2c32c49cd782451ca0cff3146ddc13fe47f415cc70b6231796a9 WatchSource:0}: Error finding container 32e7654acc7f2c32c49cd782451ca0cff3146ddc13fe47f415cc70b6231796a9: Status 404 returned error can't find the container with id 32e7654acc7f2c32c49cd782451ca0cff3146ddc13fe47f415cc70b6231796a9 Jul 28 19:37:28.342649 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.342627 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lk7lh"] Jul 28 19:37:28.352565 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:28.351882 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4cc05e_dbce_4ef2_a305_9a5a54f63e2f.slice/crio-42e9492c95061a2f86272c8f9869ca72a230d44135c660ba10f85fd40da79a30 WatchSource:0}: Error finding container 42e9492c95061a2f86272c8f9869ca72a230d44135c660ba10f85fd40da79a30: Status 404 returned error can't find the container with id 42e9492c95061a2f86272c8f9869ca72a230d44135c660ba10f85fd40da79a30 Jul 28 19:37:28.553786 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.553745 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkv6s" event={"ID":"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5","Type":"ContainerStarted","Data":"8f4b5993d962830a09e6b0a544441b6caab7cf74d228a71c37b65e1a3a611724"} Jul 28 19:37:28.557198 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.557163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" event={"ID":"89664ed0-d7e1-4548-bf7c-d923c48b3ede","Type":"ContainerStarted","Data":"59de2be8e25e9ccbc6900760509152bc87b1831624752858dc74b6f9a2be6674"} Jul 28 19:37:28.559918 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.559847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" event={"ID":"092fbc21-4f74-489b-9941-edf6694f4c23","Type":"ContainerStarted","Data":"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566"} Jul 28 19:37:28.559918 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.559888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" event={"ID":"092fbc21-4f74-489b-9941-edf6694f4c23","Type":"ContainerStarted","Data":"32e7654acc7f2c32c49cd782451ca0cff3146ddc13fe47f415cc70b6231796a9"} Jul 28 19:37:28.560105 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.559925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:28.561588 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.561504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" event={"ID":"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1","Type":"ContainerStarted","Data":"52dd433855766d845138f132f89a0a37f316a04cac92faec6ffb30be26e8a3fc"} Jul 28 19:37:28.565675 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.564942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bdcf959c-db97m" event={"ID":"630164a9-08ce-432d-ba5a-f2047a822a82","Type":"ContainerStarted","Data":"0160b1282d1b45825bf1cc01190baa5809d8d7c76ad119d7a6603cb3cf024ac9"} Jul 28 19:37:28.565675 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.564988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bdcf959c-db97m" event={"ID":"630164a9-08ce-432d-ba5a-f2047a822a82","Type":"ContainerStarted","Data":"4f25edf853b1c90192a8c0a25013c961a61cf7ddc864a35d483d16d49dd9d361"} Jul 28 19:37:28.567291 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.567017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfcxs" event={"ID":"f8843a87-0c48-40f6-bc7b-4eed6db45745","Type":"ContainerStarted","Data":"014f5a12a01b2e8f9777d4f2d48cf1452e3dfb0a553a7dbe8985a42d3e875b95"} Jul 28 19:37:28.567291 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.567063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfcxs" event={"ID":"f8843a87-0c48-40f6-bc7b-4eed6db45745","Type":"ContainerStarted","Data":"54f696facc0b840a53625c32ad74345d7d258d084a7092a515af4268b3745e2d"} Jul 28 19:37:28.567575 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.567529 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:37:28.571450 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.570934 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:37:28.571450 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.571010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" event={"ID":"dce9f717-6cac-46ae-9efc-70444d11deea","Type":"ContainerStarted","Data":"3c79ce50324bb304ea7988f8a40e44d29eaf61cce391e9d90721627f50d3c90d"} Jul 28 19:37:28.571450 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.571395 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:37:28.576153 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.575958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk7lh" event={"ID":"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f","Type":"ContainerStarted","Data":"42e9492c95061a2f86272c8f9869ca72a230d44135c660ba10f85fd40da79a30"} Jul 28 19:37:28.578158 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.578135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46x8p" event={"ID":"d96ca0b6-60e2-4e21-af8c-83ee35e99958","Type":"ContainerStarted","Data":"292a616bddb8356051d28461a3e36804e7983ad23bb51c6f40c3c6b92e5c51ed"} Jul 28 19:37:28.579705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.579681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kqqfs" event={"ID":"76938137-378f-4f48-99ab-dae7a5688d58","Type":"ContainerStarted","Data":"b9ecd31b76da874db04a7e6080586432b9be1d56aaf83ce580d812fe7b094df2"} Jul 28 19:37:28.583331 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.582331 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" podStartSLOduration=56.58231525 podStartE2EDuration="56.58231525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:28.581630167 +0000 UTC m=+68.017584159" watchObservedRunningTime="2026-07-28 19:37:28.58231525 +0000 UTC m=+68.018269206" Jul 28 19:37:28.598865 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.598217 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xfcxs" podStartSLOduration=66.598200182 podStartE2EDuration="1m6.598200182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:28.597701638 +0000 UTC m=+68.033655590" watchObservedRunningTime="2026-07-28 19:37:28.598200182 +0000 UTC m=+68.034154136" Jul 28 19:37:28.637567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.637480 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" podStartSLOduration=49.890095219 podStartE2EDuration="57.637452924s" totalImagesPullingTime="7.747357705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:28.636694961 +0000 UTC m=+68.072648914" watchObservedRunningTime="2026-07-28 19:37:28.637452924 +0000 UTC m=+68.073406881" Jul 28 19:37:28.638077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:28.638045 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7bdcf959c-db97m" podStartSLOduration=62.638036761 podStartE2EDuration="1m2.638036761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:28.618854619 +0000 UTC m=+68.054808573" watchObservedRunningTime="2026-07-28 19:37:28.638036761 +0000 UTC m=+68.073990714" Jul 28 19:37:29.045505 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.044289 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:29.048600 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.047625 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:29.404362 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.404332 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-576sb" Jul 28 19:37:29.586474 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.586375 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:29.589795 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.589549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7bdcf959c-db97m" Jul 28 19:37:29.621004 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.620038 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:37:29.677047 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.677021 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-wwv56"] Jul 28 19:37:29.692744 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.691589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:29.702236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.699017 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 19:37:29.702236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.699353 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 19:37:29.713763 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.713102 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7msvz\"" Jul 28 19:37:29.713763 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.713560 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wwv56"] Jul 28 19:37:29.779336 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.778216 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84b668485b-szs7c"] Jul 28 19:37:29.792630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.790031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjhw\" (UniqueName: \"kubernetes.io/projected/9ed56166-4e0c-4475-b141-49a89fcd2a30-kube-api-access-9hjhw\") pod \"downloads-7c97dc4555-wwv56\" (UID: \"9ed56166-4e0c-4475-b141-49a89fcd2a30\") " pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:29.809589 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.809545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b668485b-szs7c"] Jul 28 19:37:29.809732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.809678 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.891264 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.891232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjhw\" (UniqueName: \"kubernetes.io/projected/9ed56166-4e0c-4475-b141-49a89fcd2a30-kube-api-access-9hjhw\") pod \"downloads-7c97dc4555-wwv56\" (UID: \"9ed56166-4e0c-4475-b141-49a89fcd2a30\") " pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:29.907098 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.907043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjhw\" (UniqueName: \"kubernetes.io/projected/9ed56166-4e0c-4475-b141-49a89fcd2a30-kube-api-access-9hjhw\") pod \"downloads-7c97dc4555-wwv56\" (UID: \"9ed56166-4e0c-4475-b141-49a89fcd2a30\") " pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-trusted-ca\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-certificates\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-bound-sa-token\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1cc11-3e0f-4717-af58-7d8adf7a8345-ca-trust-extracted\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-tls\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-image-registry-private-configuration\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-installation-pull-secrets\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:29.992808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:29.992613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbd7\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-kube-api-access-cqbd7\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.023931 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.023736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:30.093677 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-tls\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-image-registry-private-configuration\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-installation-pull-secrets\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbd7\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-kube-api-access-cqbd7\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-trusted-ca\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-certificates\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-bound-sa-token\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.093946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1cc11-3e0f-4717-af58-7d8adf7a8345-ca-trust-extracted\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.094486 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.094360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1cc11-3e0f-4717-af58-7d8adf7a8345-ca-trust-extracted\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.097265 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.096035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-certificates\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.097870 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.097826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1cc11-3e0f-4717-af58-7d8adf7a8345-trusted-ca\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.098440 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.098392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-registry-tls\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.100034 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.099952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-image-registry-private-configuration\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.100858 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.100832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1cc11-3e0f-4717-af58-7d8adf7a8345-installation-pull-secrets\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.106522 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.106390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-bound-sa-token\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.107240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.107197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbd7\" (UniqueName: \"kubernetes.io/projected/88d1cc11-3e0f-4717-af58-7d8adf7a8345-kube-api-access-cqbd7\") pod \"image-registry-84b668485b-szs7c\" (UID: \"88d1cc11-3e0f-4717-af58-7d8adf7a8345\") " pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.136828 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.135927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:30.253506 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.253433 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wwv56"] Jul 28 19:37:30.353510 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.353485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b668485b-szs7c"] Jul 28 19:37:30.595170 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.595075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46x8p" event={"ID":"d96ca0b6-60e2-4e21-af8c-83ee35e99958","Type":"ContainerStarted","Data":"f067fb6b15396db84cd23ec645a5f1417d6833b968dbb944c0c32f027d285adb"} Jul 28 19:37:30.595170 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.595159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46x8p" event={"ID":"d96ca0b6-60e2-4e21-af8c-83ee35e99958","Type":"ContainerStarted","Data":"201d18a53143ff714897890b720d94fab50eb9a2a0904e470d95070335732a58"} Jul 28 19:37:30.640206 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:30.639557 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-46x8p" podStartSLOduration=66.86089748 podStartE2EDuration="1m8.639534446s" totalImagesPullingTime="1.778636966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:30.638177692 +0000 UTC m=+70.074131645" watchObservedRunningTime="2026-07-28 19:37:30.639534446 +0000 UTC m=+70.075488400" Jul 28 19:37:34.608394 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:34.608364 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed56166_4e0c_4475_b141_49a89fcd2a30.slice/crio-13aa5c4b6f9533775faa72e1e899faf25b48253403c5b084e90c1acabdc359f8 WatchSource:0}: Error finding container 13aa5c4b6f9533775faa72e1e899faf25b48253403c5b084e90c1acabdc359f8: Status 404 returned error can't find the container with id 13aa5c4b6f9533775faa72e1e899faf25b48253403c5b084e90c1acabdc359f8 Jul 28 19:37:35.164768 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:35.164737 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d1cc11_3e0f_4717_af58_7d8adf7a8345.slice/crio-40ce8651e4ed6dc0a2ac2596d6a1bfd8fff9ce193b08e1a454df8eef6d1901a1 WatchSource:0}: Error finding container 40ce8651e4ed6dc0a2ac2596d6a1bfd8fff9ce193b08e1a454df8eef6d1901a1: Status 404 returned error can't find the container with id 40ce8651e4ed6dc0a2ac2596d6a1bfd8fff9ce193b08e1a454df8eef6d1901a1 Jul 28 19:37:35.616989 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:35.616900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wwv56" event={"ID":"9ed56166-4e0c-4475-b141-49a89fcd2a30","Type":"ContainerStarted","Data":"13aa5c4b6f9533775faa72e1e899faf25b48253403c5b084e90c1acabdc359f8"} Jul 28 19:37:35.618378 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:35.618345 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b668485b-szs7c" event={"ID":"88d1cc11-3e0f-4717-af58-7d8adf7a8345","Type":"ContainerStarted","Data":"40ce8651e4ed6dc0a2ac2596d6a1bfd8fff9ce193b08e1a454df8eef6d1901a1"} Jul 28 19:37:37.630848 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.630760 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" event={"ID":"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1","Type":"ContainerStarted","Data":"8a70bf4b5153f0e45b5e8db93364656a7c28166bf45ed1369923f4a5269c2973"} Jul 28 19:37:37.630848 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.630837 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" event={"ID":"d101e2cd-cac4-4e7a-ba14-1a4f604f7de1","Type":"ContainerStarted","Data":"2f66b6717a34eee5ce14d744c6b8477a6e9beb2ed3ad7a64ec5f2f48ddace0b8"} Jul 28 19:37:37.633334 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.633278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk7lh" event={"ID":"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f","Type":"ContainerStarted","Data":"ea08becb2e3d36e5f6180a14ef7b95683c4067300ef1cfcf039aecd729312c29"} Jul 28 19:37:37.633621 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.633515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:37.633621 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.633550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk7lh" event={"ID":"ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f","Type":"ContainerStarted","Data":"3d45e8e10d89821c0b12540774b6beeb960dd32057fe8f683441cd6682146989"} Jul 28 19:37:37.635514 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.635470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kqqfs" event={"ID":"76938137-378f-4f48-99ab-dae7a5688d58","Type":"ContainerStarted","Data":"27c5fe4905a0b8da2378f64c7e717e36858e90a1e001ccf3b42aec7192eb80f1"} Jul 28 19:37:37.638206 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.638181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkv6s" event={"ID":"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5","Type":"ContainerStarted","Data":"0f239302c1a11aa67bea0688ef0a7994091ae1d32d635d34506076499f254f64"} Jul 28 19:37:37.640349 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.640303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" event={"ID":"89664ed0-d7e1-4548-bf7c-d923c48b3ede","Type":"ContainerStarted","Data":"7060e224b95827523e8b88761ebf1552cb74e10ff204bb61f7369b88d7a945d5"} Jul 28 19:37:37.642069 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.642038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b668485b-szs7c" event={"ID":"88d1cc11-3e0f-4717-af58-7d8adf7a8345","Type":"ContainerStarted","Data":"ca7c43b8f03f589b00b52ca0abb794c1938075a011ab65a5dd7f639648cacfcb"} Jul 28 19:37:37.642525 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.642499 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:37:37.649106 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.649056 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-n4srj" podStartSLOduration=60.306042117 podStartE2EDuration="1m8.649039289s" totalImagesPullingTime="8.342997172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:37.647406664 +0000 UTC m=+77.083360617" watchObservedRunningTime="2026-07-28 19:37:37.649039289 +0000 UTC m=+77.084993242" Jul 28 19:37:37.665632 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.665579 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lk7lh" podStartSLOduration=34.332295889 podStartE2EDuration="42.665564494s" totalImagesPullingTime="8.333268605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:37.664864112 +0000 UTC m=+77.100818071" watchObservedRunningTime="2026-07-28 19:37:37.665564494 +0000 UTC m=+77.101518449" Jul 28 19:37:37.685056 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.685015 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84b668485b-szs7c" podStartSLOduration=8.684999525 podStartE2EDuration="8.684999525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:37.684195981 +0000 UTC m=+77.120149937" watchObservedRunningTime="2026-07-28 19:37:37.684999525 +0000 UTC m=+77.120953481" Jul 28 19:37:37.700832 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.700787 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tbm99" podStartSLOduration=63.235547757 podStartE2EDuration="1m11.700770763s" totalImagesPullingTime="8.465223006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:37.699864695 +0000 UTC m=+77.135818649" watchObservedRunningTime="2026-07-28 19:37:37.700770763 +0000 UTC m=+77.136724718" Jul 28 19:37:37.714449 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:37.714407 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kqqfs" podStartSLOduration=66.722019566 podStartE2EDuration="1m15.714391505s" totalImagesPullingTime="8.992371939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:37.713856634 +0000 UTC m=+77.149810587" watchObservedRunningTime="2026-07-28 19:37:37.714391505 +0000 UTC m=+77.150345458" Jul 28 19:37:41.664909 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:41.664873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkv6s" event={"ID":"86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5","Type":"ContainerStarted","Data":"f3e12ee6dfe7c38382723c766f0ea73fdf883bd115ecd8ebb7e7a76db37c7b84"} Jul 28 19:37:41.683714 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:41.683645 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkv6s" podStartSLOduration=17.44238455 podStartE2EDuration="35.683623559s" totalImagesPullingTime="18.241239009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:37:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:41.681518711 +0000 UTC m=+81.117472667" watchObservedRunningTime="2026-07-28 19:37:41.683623559 +0000 UTC m=+81.119577515" Jul 28 19:37:47.650173 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:47.650050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lk7lh" Jul 28 19:37:50.603781 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.603747 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:50.939717 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.939681 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z8xzf"] Jul 28 19:37:50.987236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.987208 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:50.990132 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.990104 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 19:37:50.990371 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.990356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 19:37:50.991088 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.991068 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pwz2m\"" Jul 28 19:37:50.991193 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.991098 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 19:37:50.991193 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:50.991140 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 19:37:51.076359 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tmc\" (UniqueName: \"kubernetes.io/projected/67cd925f-253f-4df9-b7f0-53a645bf0386-kube-api-access-44tmc\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-tls\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-metrics-client-ca\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-wtmp\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076692 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-textfile\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076692 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-sys\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.076778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.076722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-root\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178262 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-root\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178262 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tmc\" (UniqueName: \"kubernetes.io/projected/67cd925f-253f-4df9-b7f0-53a645bf0386-kube-api-access-44tmc\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178262 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-tls\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178262 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-root\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-metrics-client-ca\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-wtmp\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178392 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-textfile\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-sys\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178753 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-sys\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178753 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-wtmp\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.178943 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.178924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-textfile\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.179607 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.179586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.179837 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.179810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cd925f-253f-4df9-b7f0-53a645bf0386-metrics-client-ca\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.181056 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.181031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-tls\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.181171 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.181043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67cd925f-253f-4df9-b7f0-53a645bf0386-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.189544 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.189506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tmc\" (UniqueName: \"kubernetes.io/projected/67cd925f-253f-4df9-b7f0-53a645bf0386-kube-api-access-44tmc\") pod \"node-exporter-z8xzf\" (UID: \"67cd925f-253f-4df9-b7f0-53a645bf0386\") " pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:51.297493 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:51.297417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8xzf" Jul 28 19:37:52.077161 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:37:52.077127 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cd925f_253f_4df9_b7f0_53a645bf0386.slice/crio-ee5abac892d3d7fbe42aafff3ae61b24c56e9add2c9fbf2269c15ba01625e82c WatchSource:0}: Error finding container ee5abac892d3d7fbe42aafff3ae61b24c56e9add2c9fbf2269c15ba01625e82c: Status 404 returned error can't find the container with id ee5abac892d3d7fbe42aafff3ae61b24c56e9add2c9fbf2269c15ba01625e82c Jul 28 19:37:52.710547 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.710493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8xzf" event={"ID":"67cd925f-253f-4df9-b7f0-53a645bf0386","Type":"ContainerStarted","Data":"ee5abac892d3d7fbe42aafff3ae61b24c56e9add2c9fbf2269c15ba01625e82c"} Jul 28 19:37:52.712986 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.712930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wwv56" event={"ID":"9ed56166-4e0c-4475-b141-49a89fcd2a30","Type":"ContainerStarted","Data":"df42bcf894dc47b6bcbb208c4f2b87befc9f131b62813f1479024991235adbd5"} Jul 28 19:37:52.713246 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.713214 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:52.715069 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.715035 2562 patch_prober.go:28] interesting pod/downloads-7c97dc4555-wwv56 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.19:8080/\": dial tcp 10.132.0.19:8080: connect: connection refused" start-of-body= Jul 28 19:37:52.715171 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.715093 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-wwv56" podUID="9ed56166-4e0c-4475-b141-49a89fcd2a30" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.19:8080/\": dial tcp 10.132.0.19:8080: connect: connection refused" Jul 28 19:37:52.734282 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:52.734240 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-wwv56" podStartSLOduration=6.042032247 podStartE2EDuration="23.734227762s" totalImagesPullingTime="17.692195515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:52.732183977 +0000 UTC m=+92.168137931" watchObservedRunningTime="2026-07-28 19:37:52.734227762 +0000 UTC m=+92.170181726" Jul 28 19:37:53.719616 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:53.719576 2562 generic.go:356] "Generic (PLEG): container finished" podID="67cd925f-253f-4df9-b7f0-53a645bf0386" containerID="ba9be386f5f43d1b56740c41f4dca6e26dff347c0d7806309dfbab24c6b0a0c9" exitCode=0 Jul 28 19:37:53.720139 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:53.719694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8xzf" event={"ID":"67cd925f-253f-4df9-b7f0-53a645bf0386","Type":"ContainerDied","Data":"ba9be386f5f43d1b56740c41f4dca6e26dff347c0d7806309dfbab24c6b0a0c9"} Jul 28 19:37:53.731786 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:53.731762 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-wwv56" Jul 28 19:37:54.727549 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:54.727507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8xzf" event={"ID":"67cd925f-253f-4df9-b7f0-53a645bf0386","Type":"ContainerStarted","Data":"0f5bb6f74a75239aa581e398d4336c24bb5a3c98d233b25b6c2365cde273627e"} Jul 28 19:37:54.728040 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:54.727572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8xzf" event={"ID":"67cd925f-253f-4df9-b7f0-53a645bf0386","Type":"ContainerStarted","Data":"72290386cac595ae592edfe56bffdd88bbbefbff06b13354fdb7c9bae5fbc32b"} Jul 28 19:37:54.748708 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:54.748649 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z8xzf" podStartSLOduration=3.663938078 podStartE2EDuration="4.748630813s" totalImagesPullingTime="1.002146751s" totalInitContainerRuntime="82.545984ms" isStatefulPod=true podCreationTimestamp="2026-07-28 19:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:54.746308416 +0000 UTC m=+94.182262392" watchObservedRunningTime="2026-07-28 19:37:54.748630813 +0000 UTC m=+94.184584766" Jul 28 19:37:55.626598 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:55.626497 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" podUID="092fbc21-4f74-489b-9941-edf6694f4c23" containerName="registry" containerID="cri-o://74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566" gracePeriod=30 Jul 28 19:37:55.923696 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:55.923642 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:56.021165 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021136 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021343 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021191 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021343 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021223 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021424 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnj2b\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021570 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021515 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021570 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021556 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021566 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 19:37:56.021686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021594 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration\") pod \"092fbc21-4f74-489b-9941-edf6694f4c23\" (UID: \"092fbc21-4f74-489b-9941-edf6694f4c23\") " Jul 28 19:37:56.021686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021573 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 19:37:56.022318 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021893 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-registry-certificates\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.022318 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.021917 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/092fbc21-4f74-489b-9941-edf6694f4c23-trusted-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.024269 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.024164 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b" (OuterVolumeSpecName: "kube-api-access-wnj2b") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "kube-api-access-wnj2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:37:56.024429 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.024281 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:37:56.024591 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.024562 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:37:56.024715 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.024570 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:37:56.024715 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.024645 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:37:56.034508 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.034478 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "092fbc21-4f74-489b-9941-edf6694f4c23" (UID: "092fbc21-4f74-489b-9941-edf6694f4c23"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:37:56.122530 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122489 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/092fbc21-4f74-489b-9941-edf6694f4c23-ca-trust-extracted\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.122530 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122528 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnj2b\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-kube-api-access-wnj2b\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.122761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122540 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-registry-tls\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.122761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122548 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/092fbc21-4f74-489b-9941-edf6694f4c23-bound-sa-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.122761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122580 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-image-registry-private-configuration\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.122761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.122591 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/092fbc21-4f74-489b-9941-edf6694f4c23-installation-pull-secrets\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:37:56.743707 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.743639 2562 generic.go:356] "Generic (PLEG): container finished" podID="092fbc21-4f74-489b-9941-edf6694f4c23" containerID="74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566" exitCode=0 Jul 28 19:37:56.743907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.743706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" event={"ID":"092fbc21-4f74-489b-9941-edf6694f4c23","Type":"ContainerDied","Data":"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566"} Jul 28 19:37:56.743907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.743719 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" Jul 28 19:37:56.743907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.743824 2562 scope.go:117] "RemoveContainer" containerID="74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566" Jul 28 19:37:56.743907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.743809 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cbbfd6c4c-csfq7" event={"ID":"092fbc21-4f74-489b-9941-edf6694f4c23","Type":"ContainerDied","Data":"32e7654acc7f2c32c49cd782451ca0cff3146ddc13fe47f415cc70b6231796a9"} Jul 28 19:37:56.753972 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.753954 2562 scope.go:117] "RemoveContainer" containerID="74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566" Jul 28 19:37:56.754273 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:37:56.754249 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566\": container with ID starting with 74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566 not found: ID does not exist" containerID="74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566" Jul 28 19:37:56.754347 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.754280 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566"} err="failed to get container status \"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566\": rpc error: code = NotFound desc = could not find container \"74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566\": container with ID starting with 74b36a18432f3c4040f12ed09cb0d33f595f3de98a47e4f0a5b6a9cb3dabb566 not found: ID does not exist" Jul 28 19:37:56.772615 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.772590 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:37:56.791107 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:56.791080 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cbbfd6c4c-csfq7"] Jul 28 19:37:57.162157 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:57.162125 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092fbc21-4f74-489b-9941-edf6694f4c23" path="/var/lib/kubelet/pods/092fbc21-4f74-489b-9941-edf6694f4c23/volumes" Jul 28 19:37:59.659163 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:37:59.659135 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84b668485b-szs7c" Jul 28 19:38:00.599276 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:00.599209 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xfcxs" Jul 28 19:38:16.818363 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:16.818326 2562 generic.go:356] "Generic (PLEG): container finished" podID="f4c57e53-4a0a-4b42-ade0-44ed7a500ed7" containerID="f55bd36daef3cbf814d7c36a310db52a12c4c55b64daf3ec0b9f668a648e2e93" exitCode=0 Jul 28 19:38:16.818791 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:16.818370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dvczk" event={"ID":"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7","Type":"ContainerDied","Data":"f55bd36daef3cbf814d7c36a310db52a12c4c55b64daf3ec0b9f668a648e2e93"} Jul 28 19:38:16.818791 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:16.818689 2562 scope.go:117] "RemoveContainer" containerID="f55bd36daef3cbf814d7c36a310db52a12c4c55b64daf3ec0b9f668a648e2e93" Jul 28 19:38:17.823777 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:17.823737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dvczk" event={"ID":"f4c57e53-4a0a-4b42-ade0-44ed7a500ed7","Type":"ContainerStarted","Data":"b674e97b13a1892aa6a08adc6b24a41c3e1269b25ba0b96901a783ce34afff54"} Jul 28 19:38:26.856029 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:26.855959 2562 generic.go:356] "Generic (PLEG): container finished" podID="45bcbaaa-d4fd-4c39-89fb-3c05b8102493" containerID="724aabafab3577a3c3651ba9dcb7cfed3882fcbf1c0f3fee189dd56da2d774ca" exitCode=0 Jul 28 19:38:26.856029 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:26.855994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" event={"ID":"45bcbaaa-d4fd-4c39-89fb-3c05b8102493","Type":"ContainerDied","Data":"724aabafab3577a3c3651ba9dcb7cfed3882fcbf1c0f3fee189dd56da2d774ca"} Jul 28 19:38:26.856421 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:26.856280 2562 scope.go:117] "RemoveContainer" containerID="724aabafab3577a3c3651ba9dcb7cfed3882fcbf1c0f3fee189dd56da2d774ca" Jul 28 19:38:27.861730 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:27.861693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g" event={"ID":"45bcbaaa-d4fd-4c39-89fb-3c05b8102493","Type":"ContainerStarted","Data":"d649e6480df5ccc6d283e5f4f50464492325161c1edc8cda5d59bf9a80d8347b"} Jul 28 19:38:35.896479 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:35.896442 2562 generic.go:356] "Generic (PLEG): container finished" podID="92eb4695-0595-47ed-8a51-ad1eff1364eb" containerID="76c138cb4cc927be6cbfbd745b731a48604d25a31ec563f7858365215fb807d6" exitCode=0 Jul 28 19:38:35.897018 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:35.896518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" event={"ID":"92eb4695-0595-47ed-8a51-ad1eff1364eb","Type":"ContainerDied","Data":"76c138cb4cc927be6cbfbd745b731a48604d25a31ec563f7858365215fb807d6"} Jul 28 19:38:35.897018 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:35.896921 2562 scope.go:117] "RemoveContainer" containerID="76c138cb4cc927be6cbfbd745b731a48604d25a31ec563f7858365215fb807d6" Jul 28 19:38:36.901557 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:38:36.901526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zbr5p" event={"ID":"92eb4695-0595-47ed-8a51-ad1eff1364eb","Type":"ContainerStarted","Data":"1926f7fa382835f9be49f521b159ab8cff6ef608d082ea8fb8c546733441d2e7"} Jul 28 19:39:15.239866 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.239786 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69f797dc9d-zsslk"] Jul 28 19:39:15.240296 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.240094 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092fbc21-4f74-489b-9941-edf6694f4c23" containerName="registry" Jul 28 19:39:15.240296 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.240105 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="092fbc21-4f74-489b-9941-edf6694f4c23" containerName="registry" Jul 28 19:39:15.240296 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.240160 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="092fbc21-4f74-489b-9941-edf6694f4c23" containerName="registry" Jul 28 19:39:15.242341 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.242316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.244962 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.244937 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 19:39:15.245079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.244990 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 19:39:15.245079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.245045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n4v5s\"" Jul 28 19:39:15.245079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.245072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 19:39:15.245323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.245302 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 19:39:15.245425 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.245403 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 19:39:15.251194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.251172 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 19:39:15.259330 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.259307 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69f797dc9d-zsslk"] Jul 28 19:39:15.327530 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327530 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmzf\" (UniqueName: \"kubernetes.io/projected/2cf68961-85dc-4df6-af7e-9a81b397e9be-kube-api-access-rmmzf\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-serving-certs-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-federate-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-metrics-client-ca\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.327799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.327708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.428968 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.428928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.428968 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.428970 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmzf\" (UniqueName: \"kubernetes.io/projected/2cf68961-85dc-4df6-af7e-9a81b397e9be-kube-api-access-rmmzf\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-serving-certs-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-federate-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429356 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-metrics-client-ca\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429356 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429356 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.429946 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.429919 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-serving-certs-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.430131 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.430011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-metrics-client-ca\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.430275 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.430254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.431612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.431587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.431730 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.431709 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.431939 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.431920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-telemeter-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.432048 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.432030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2cf68961-85dc-4df6-af7e-9a81b397e9be-federate-client-tls\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.436941 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.436922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmzf\" (UniqueName: \"kubernetes.io/projected/2cf68961-85dc-4df6-af7e-9a81b397e9be-kube-api-access-rmmzf\") pod \"telemeter-client-69f797dc9d-zsslk\" (UID: \"2cf68961-85dc-4df6-af7e-9a81b397e9be\") " pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.551935 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.551843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" Jul 28 19:39:15.701742 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:15.701719 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69f797dc9d-zsslk"] Jul 28 19:39:15.703928 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:39:15.703900 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf68961_85dc_4df6_af7e_9a81b397e9be.slice/crio-cb9f828a891208c0014ce5fc0b14e49e6d376a5a6725558973b9707dae30938d WatchSource:0}: Error finding container cb9f828a891208c0014ce5fc0b14e49e6d376a5a6725558973b9707dae30938d: Status 404 returned error can't find the container with id cb9f828a891208c0014ce5fc0b14e49e6d376a5a6725558973b9707dae30938d Jul 28 19:39:16.037015 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:16.036975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" event={"ID":"2cf68961-85dc-4df6-af7e-9a81b397e9be","Type":"ContainerStarted","Data":"cb9f828a891208c0014ce5fc0b14e49e6d376a5a6725558973b9707dae30938d"} Jul 28 19:39:18.045513 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:18.045476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" event={"ID":"2cf68961-85dc-4df6-af7e-9a81b397e9be","Type":"ContainerStarted","Data":"f350e82a09576eef76a7410d4289fc315f2f5b70d9aa3ac09f053cb7c66572cf"} Jul 28 19:39:19.051267 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:19.051231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" event={"ID":"2cf68961-85dc-4df6-af7e-9a81b397e9be","Type":"ContainerStarted","Data":"84a2819af345c928e83ad957602165cfff38a4d46a91f9f7849af8d0dc1134a1"} Jul 28 19:39:19.051757 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:19.051286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" event={"ID":"2cf68961-85dc-4df6-af7e-9a81b397e9be","Type":"ContainerStarted","Data":"49131949d4088125d7ea1ae06e82e3120cf137c696f73948b187281a7ccd2d50"} Jul 28 19:39:19.074223 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:39:19.074166 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69f797dc9d-zsslk" podStartSLOduration=1.2549027289999999 podStartE2EDuration="4.074146039s" totalImagesPullingTime="2.81924331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:39:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:39:19.071606168 +0000 UTC m=+178.507560120" watchObservedRunningTime="2026-07-28 19:39:19.074146039 +0000 UTC m=+178.510099995" Jul 28 19:41:17.958319 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.958286 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c"] Jul 28 19:41:17.961539 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.961521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:17.964256 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.964236 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 19:41:17.964387 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.964240 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 19:41:17.965270 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.965249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v6fq6\"" Jul 28 19:41:17.970872 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:17.970850 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c"] Jul 28 19:41:18.039229 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.039208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.039336 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.039237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dftq\" (UniqueName: \"kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.039336 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.039282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.139971 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.139945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.140088 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.139989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.140088 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.140007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dftq\" (UniqueName: \"kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.140314 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.140295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.140378 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.140334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.148543 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.148516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dftq\" (UniqueName: \"kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.271204 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.271147 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:18.395599 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.395571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c"] Jul 28 19:41:18.398109 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:41:18.398083 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0151a8_684f_4b9c_8a91_15f710e46c98.slice/crio-75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0 WatchSource:0}: Error finding container 75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0: Status 404 returned error can't find the container with id 75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0 Jul 28 19:41:18.475058 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:18.475033 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" event={"ID":"4d0151a8-684f-4b9c-8a91-15f710e46c98","Type":"ContainerStarted","Data":"75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0"} Jul 28 19:41:20.985509 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:20.985476 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:41:20.991855 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:20.991831 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:41:20.998993 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:20.998973 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 19:41:25.504623 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:25.504534 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerID="9f4a9cf9d015617f6b52db744d5e1e856ac8663c08e4bd986d1472b5ef234ffb" exitCode=0 Jul 28 19:41:25.505019 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:25.504616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" event={"ID":"4d0151a8-684f-4b9c-8a91-15f710e46c98","Type":"ContainerDied","Data":"9f4a9cf9d015617f6b52db744d5e1e856ac8663c08e4bd986d1472b5ef234ffb"} Jul 28 19:41:25.505563 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:25.505547 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 19:41:30.531024 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:30.530987 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerID="b3956d43ce991209d403f0f842084b79f3868bc8c737092d28db0da720d5a5f0" exitCode=0 Jul 28 19:41:30.531450 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:30.531076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" event={"ID":"4d0151a8-684f-4b9c-8a91-15f710e46c98","Type":"ContainerDied","Data":"b3956d43ce991209d403f0f842084b79f3868bc8c737092d28db0da720d5a5f0"} Jul 28 19:41:37.562269 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:37.562233 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerID="1c2b044bddcc1ccc3a5ec52e372fbdfba89de882bd1b6cb63b2649bf2941f291" exitCode=0 Jul 28 19:41:37.562709 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:37.562297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" event={"ID":"4d0151a8-684f-4b9c-8a91-15f710e46c98","Type":"ContainerDied","Data":"1c2b044bddcc1ccc3a5ec52e372fbdfba89de882bd1b6cb63b2649bf2941f291"} Jul 28 19:41:38.697187 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.697165 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:38.801872 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.801831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dftq\" (UniqueName: \"kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq\") pod \"4d0151a8-684f-4b9c-8a91-15f710e46c98\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " Jul 28 19:41:38.801872 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.801880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle\") pod \"4d0151a8-684f-4b9c-8a91-15f710e46c98\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " Jul 28 19:41:38.802059 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.801956 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util\") pod \"4d0151a8-684f-4b9c-8a91-15f710e46c98\" (UID: \"4d0151a8-684f-4b9c-8a91-15f710e46c98\") " Jul 28 19:41:38.802557 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.802527 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle" (OuterVolumeSpecName: "bundle") pod "4d0151a8-684f-4b9c-8a91-15f710e46c98" (UID: "4d0151a8-684f-4b9c-8a91-15f710e46c98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:41:38.803979 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.803954 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq" (OuterVolumeSpecName: "kube-api-access-5dftq") pod "4d0151a8-684f-4b9c-8a91-15f710e46c98" (UID: "4d0151a8-684f-4b9c-8a91-15f710e46c98"). InnerVolumeSpecName "kube-api-access-5dftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:41:38.806185 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.806162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util" (OuterVolumeSpecName: "util") pod "4d0151a8-684f-4b9c-8a91-15f710e46c98" (UID: "4d0151a8-684f-4b9c-8a91-15f710e46c98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:41:38.903037 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.903012 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:38.903182 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.903039 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dftq\" (UniqueName: \"kubernetes.io/projected/4d0151a8-684f-4b9c-8a91-15f710e46c98-kube-api-access-5dftq\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:38.903182 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:38.903056 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d0151a8-684f-4b9c-8a91-15f710e46c98-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:39.570864 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:39.570799 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" Jul 28 19:41:39.570864 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:39.570813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qr75c" event={"ID":"4d0151a8-684f-4b9c-8a91-15f710e46c98","Type":"ContainerDied","Data":"75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0"} Jul 28 19:41:39.571036 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:39.570877 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a2cc8b077769836333528853a0bde63931ba20684f38ef64dba703c0283ac0" Jul 28 19:41:46.016470 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016434 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747"] Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016764 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="util" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016775 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="util" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016784 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="extract" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016790 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="extract" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016797 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="pull" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016803 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="pull" Jul 28 19:41:46.016927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.016852 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0151a8-684f-4b9c-8a91-15f710e46c98" containerName="extract" Jul 28 19:41:46.019716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.019700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.024631 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.024611 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k92fz\"" Jul 28 19:41:46.024958 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.024937 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:41:46.025311 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.025296 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 19:41:46.034335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.034316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747"] Jul 28 19:41:46.061406 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.061384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896641db-e0d3-4d92-a585-38d1304d899b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.061506 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.061431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6bp\" (UniqueName: \"kubernetes.io/projected/896641db-e0d3-4d92-a585-38d1304d899b-kube-api-access-dm6bp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.162459 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.162435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896641db-e0d3-4d92-a585-38d1304d899b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.162555 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.162476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6bp\" (UniqueName: \"kubernetes.io/projected/896641db-e0d3-4d92-a585-38d1304d899b-kube-api-access-dm6bp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.162804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.162787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896641db-e0d3-4d92-a585-38d1304d899b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.171490 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.171472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6bp\" (UniqueName: \"kubernetes.io/projected/896641db-e0d3-4d92-a585-38d1304d899b-kube-api-access-dm6bp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6z747\" (UID: \"896641db-e0d3-4d92-a585-38d1304d899b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.329092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.329031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" Jul 28 19:41:46.457310 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.457289 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747"] Jul 28 19:41:46.459962 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:41:46.459933 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896641db_e0d3_4d92_a585_38d1304d899b.slice/crio-13bb3fda5604e7935b1056a01418748c7d8697158f58c14c6a7356149bdea689 WatchSource:0}: Error finding container 13bb3fda5604e7935b1056a01418748c7d8697158f58c14c6a7356149bdea689: Status 404 returned error can't find the container with id 13bb3fda5604e7935b1056a01418748c7d8697158f58c14c6a7356149bdea689 Jul 28 19:41:46.597605 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:46.597529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" event={"ID":"896641db-e0d3-4d92-a585-38d1304d899b","Type":"ContainerStarted","Data":"13bb3fda5604e7935b1056a01418748c7d8697158f58c14c6a7356149bdea689"} Jul 28 19:41:48.608282 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:48.608254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" event={"ID":"896641db-e0d3-4d92-a585-38d1304d899b","Type":"ContainerStarted","Data":"baf2f22301128937a1577e9b578a2bae1a4a30f3c7c27b0926a1da48cdbea73c"} Jul 28 19:41:48.630633 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:48.630536 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6z747" podStartSLOduration=1.558991579 podStartE2EDuration="3.630519797s" totalImagesPullingTime="2.071528218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:41:48.62978596 +0000 UTC m=+328.065739914" watchObservedRunningTime="2026-07-28 19:41:48.630519797 +0000 UTC m=+328.066473748" Jul 28 19:41:49.726517 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.726482 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l"] Jul 28 19:41:49.729901 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.729885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.732544 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.732519 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 19:41:49.732690 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.732595 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v6fq6\"" Jul 28 19:41:49.732762 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.732703 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 19:41:49.739913 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.739884 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l"] Jul 28 19:41:49.793567 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.793543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlpn\" (UniqueName: \"kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.793694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.793591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.793694 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.793679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.894612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.894579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.894757 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.894632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.894757 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.894708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlpn\" (UniqueName: \"kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.894988 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.894971 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.895059 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.895030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:49.904172 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:49.904146 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlpn\" (UniqueName: \"kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:50.039276 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:50.039221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:50.168144 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:50.168122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l"] Jul 28 19:41:50.170871 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:41:50.170842 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c31cc32_794e_434f_832d_a9698bc2eb0b.slice/crio-97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4 WatchSource:0}: Error finding container 97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4: Status 404 returned error can't find the container with id 97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4 Jul 28 19:41:50.617583 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:50.617551 2562 generic.go:356] "Generic (PLEG): container finished" podID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerID="2e760f0100259aec352fa964eb2dc300980856b04b50417388e2164dfee029bd" exitCode=0 Jul 28 19:41:50.617759 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:50.617601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" event={"ID":"0c31cc32-794e-434f-832d-a9698bc2eb0b","Type":"ContainerDied","Data":"2e760f0100259aec352fa964eb2dc300980856b04b50417388e2164dfee029bd"} Jul 28 19:41:50.617759 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:50.617644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" event={"ID":"0c31cc32-794e-434f-832d-a9698bc2eb0b","Type":"ContainerStarted","Data":"97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4"} Jul 28 19:41:53.633935 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:53.633900 2562 generic.go:356] "Generic (PLEG): container finished" podID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerID="df200a750c3d7ece085007346802eac14f83b80c1c0636d13edfa692336a2b9a" exitCode=0 Jul 28 19:41:53.633935 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:53.633938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" event={"ID":"0c31cc32-794e-434f-832d-a9698bc2eb0b","Type":"ContainerDied","Data":"df200a750c3d7ece085007346802eac14f83b80c1c0636d13edfa692336a2b9a"} Jul 28 19:41:54.641487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:54.641450 2562 generic.go:356] "Generic (PLEG): container finished" podID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerID="e6d6f2b7c15454e72b6d8d31e7b58497a060ed0795ba2fcae0c04ab249369601" exitCode=0 Jul 28 19:41:54.641884 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:54.641492 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" event={"ID":"0c31cc32-794e-434f-832d-a9698bc2eb0b","Type":"ContainerDied","Data":"e6d6f2b7c15454e72b6d8d31e7b58497a060ed0795ba2fcae0c04ab249369601"} Jul 28 19:41:55.768174 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.768143 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:55.837799 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.837773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util\") pod \"0c31cc32-794e-434f-832d-a9698bc2eb0b\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " Jul 28 19:41:55.837907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.837808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnlpn\" (UniqueName: \"kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn\") pod \"0c31cc32-794e-434f-832d-a9698bc2eb0b\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " Jul 28 19:41:55.837907 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.837879 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle\") pod \"0c31cc32-794e-434f-832d-a9698bc2eb0b\" (UID: \"0c31cc32-794e-434f-832d-a9698bc2eb0b\") " Jul 28 19:41:55.838295 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.838264 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle" (OuterVolumeSpecName: "bundle") pod "0c31cc32-794e-434f-832d-a9698bc2eb0b" (UID: "0c31cc32-794e-434f-832d-a9698bc2eb0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:41:55.840252 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.840229 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn" (OuterVolumeSpecName: "kube-api-access-pnlpn") pod "0c31cc32-794e-434f-832d-a9698bc2eb0b" (UID: "0c31cc32-794e-434f-832d-a9698bc2eb0b"). InnerVolumeSpecName "kube-api-access-pnlpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:41:55.842459 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.842436 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util" (OuterVolumeSpecName: "util") pod "0c31cc32-794e-434f-832d-a9698bc2eb0b" (UID: "0c31cc32-794e-434f-832d-a9698bc2eb0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:41:55.939051 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.939017 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:55.939051 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.939048 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c31cc32-794e-434f-832d-a9698bc2eb0b-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:55.939234 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:55.939066 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnlpn\" (UniqueName: \"kubernetes.io/projected/0c31cc32-794e-434f-832d-a9698bc2eb0b-kube-api-access-pnlpn\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:41:56.226608 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226533 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gz8vl"] Jul 28 19:41:56.226859 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226846 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="pull" Jul 28 19:41:56.226908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226861 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="pull" Jul 28 19:41:56.226908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226874 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="extract" Jul 28 19:41:56.226908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226880 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="extract" Jul 28 19:41:56.226908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226889 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="util" Jul 28 19:41:56.226908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226895 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="util" Jul 28 19:41:56.227051 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.226944 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c31cc32-794e-434f-832d-a9698bc2eb0b" containerName="extract" Jul 28 19:41:56.230702 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.230686 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.233053 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.233029 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 19:41:56.233184 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.233082 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 19:41:56.233184 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.233158 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nf7kn\"" Jul 28 19:41:56.237851 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.237832 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gz8vl"] Jul 28 19:41:56.342067 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.342040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbd5j\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-kube-api-access-pbd5j\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.342207 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.342101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.443320 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.443290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbd5j\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-kube-api-access-pbd5j\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.443439 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.443365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.453129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.453100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbd5j\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-kube-api-access-pbd5j\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.453129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.453110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ed6276-11ef-41e5-ac3f-f7ede357be7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gz8vl\" (UID: \"f1ed6276-11ef-41e5-ac3f-f7ede357be7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.551979 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.551922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:56.654566 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.654529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" event={"ID":"0c31cc32-794e-434f-832d-a9698bc2eb0b","Type":"ContainerDied","Data":"97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4"} Jul 28 19:41:56.654566 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.654567 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2tw8l" Jul 28 19:41:56.654801 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.654587 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97020aec87db1811349494d34c26502b7ffcb9e61e38344411650185175eaeb4" Jul 28 19:41:56.719720 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:56.719697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gz8vl"] Jul 28 19:41:56.721329 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:41:56.721309 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ed6276_11ef_41e5_ac3f_f7ede357be7c.slice/crio-49a1d5e6424741f4dc125c7c03d976ce09f70b574730d31515472f864b27a033 WatchSource:0}: Error finding container 49a1d5e6424741f4dc125c7c03d976ce09f70b574730d31515472f864b27a033: Status 404 returned error can't find the container with id 49a1d5e6424741f4dc125c7c03d976ce09f70b574730d31515472f864b27a033 Jul 28 19:41:57.660019 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:57.659982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" event={"ID":"f1ed6276-11ef-41e5-ac3f-f7ede357be7c","Type":"ContainerStarted","Data":"49a1d5e6424741f4dc125c7c03d976ce09f70b574730d31515472f864b27a033"} Jul 28 19:41:59.679171 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:59.679133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" event={"ID":"f1ed6276-11ef-41e5-ac3f-f7ede357be7c","Type":"ContainerStarted","Data":"d337b6f1e3a7f573a45700544b540408ebeeadf1f17cbfe09a07ff4a6379fa7e"} Jul 28 19:41:59.679509 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:59.679213 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:41:59.698288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:41:59.698243 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" podStartSLOduration=0.831968662 podStartE2EDuration="3.69823057s" totalImagesPullingTime="2.866261908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:41:59.697252107 +0000 UTC m=+339.133206058" watchObservedRunningTime="2026-07-28 19:41:59.69823057 +0000 UTC m=+339.134184521" Jul 28 19:42:05.686607 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:05.686576 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gz8vl" Jul 28 19:42:09.528803 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.528771 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt"] Jul 28 19:42:09.545007 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.544970 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt"] Jul 28 19:42:09.545162 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.545085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.548001 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.547979 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 19:42:09.548916 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.548897 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 19:42:09.549000 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.548952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v6fq6\"" Jul 28 19:42:09.646495 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.646460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmqj\" (UniqueName: \"kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.646627 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.646510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.646627 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.646572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.747524 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.747496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.747695 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.747547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.747695 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.747623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmqj\" (UniqueName: \"kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.747904 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.747883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.747962 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.747923 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.756781 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.756752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmqj\" (UniqueName: \"kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.856964 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.856913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:09.994046 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:09.994017 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt"] Jul 28 19:42:09.996641 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:09.996617 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc70e61a_9361_4a3b_b3ee_265f924056d0.slice/crio-37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b WatchSource:0}: Error finding container 37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b: Status 404 returned error can't find the container with id 37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b Jul 28 19:42:10.035200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.035177 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nb4mb"] Jul 28 19:42:10.064998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.064975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nb4mb"] Jul 28 19:42:10.065104 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.065089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.067953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.067931 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lqtgh\"" Jul 28 19:42:10.150036 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.150008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxh6\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-kube-api-access-hxxh6\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.150150 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.150041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-bound-sa-token\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.250460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.250430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxh6\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-kube-api-access-hxxh6\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.250586 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.250465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-bound-sa-token\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.260024 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.260003 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-bound-sa-token\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.273948 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.273927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxh6\" (UniqueName: \"kubernetes.io/projected/5dbac66e-5f79-4277-948e-d48945424d62-kube-api-access-hxxh6\") pod \"cert-manager-55b4d954d5-nb4mb\" (UID: \"5dbac66e-5f79-4277-948e-d48945424d62\") " pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.374949 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.374927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nb4mb" Jul 28 19:42:10.509123 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.509050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nb4mb"] Jul 28 19:42:10.511570 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:10.511539 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dbac66e_5f79_4277_948e_d48945424d62.slice/crio-c674684b9dd57638189c6d4d951b5d53a00d4850bfed34ad0df02f4850404ef5 WatchSource:0}: Error finding container c674684b9dd57638189c6d4d951b5d53a00d4850bfed34ad0df02f4850404ef5: Status 404 returned error can't find the container with id c674684b9dd57638189c6d4d951b5d53a00d4850bfed34ad0df02f4850404ef5 Jul 28 19:42:10.724061 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.724028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nb4mb" event={"ID":"5dbac66e-5f79-4277-948e-d48945424d62","Type":"ContainerStarted","Data":"e535b99e38cd020394957a9d7313d45df9493d6f9336b0dda2e907aade72680a"} Jul 28 19:42:10.724472 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.724069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nb4mb" event={"ID":"5dbac66e-5f79-4277-948e-d48945424d62","Type":"ContainerStarted","Data":"c674684b9dd57638189c6d4d951b5d53a00d4850bfed34ad0df02f4850404ef5"} Jul 28 19:42:10.725501 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.725478 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerID="2c7fd2c34cb990e0e4388129af7f9fecc65a82797afa371b7fb4f62abdc3f98b" exitCode=0 Jul 28 19:42:10.725611 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.725508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" event={"ID":"cc70e61a-9361-4a3b-b3ee-265f924056d0","Type":"ContainerDied","Data":"2c7fd2c34cb990e0e4388129af7f9fecc65a82797afa371b7fb4f62abdc3f98b"} Jul 28 19:42:10.725611 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.725551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" event={"ID":"cc70e61a-9361-4a3b-b3ee-265f924056d0","Type":"ContainerStarted","Data":"37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b"} Jul 28 19:42:10.741465 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:10.741414 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nb4mb" podStartSLOduration=0.741397685 podStartE2EDuration="741.397685ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:10.739849568 +0000 UTC m=+350.175803521" watchObservedRunningTime="2026-07-28 19:42:10.741397685 +0000 UTC m=+350.177351639" Jul 28 19:42:11.731062 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:11.731038 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerID="b8a5dc677bad10c350d5a940f27911d332d2ee7477b409f9440c954c1203be3c" exitCode=0 Jul 28 19:42:11.731470 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:11.731129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" event={"ID":"cc70e61a-9361-4a3b-b3ee-265f924056d0","Type":"ContainerDied","Data":"b8a5dc677bad10c350d5a940f27911d332d2ee7477b409f9440c954c1203be3c"} Jul 28 19:42:12.736984 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:12.736947 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerID="bcceb4fcce53f9888f6adc1c63eb6595768999b729cd5f864e753510fa7936ae" exitCode=0 Jul 28 19:42:12.737392 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:12.737026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" event={"ID":"cc70e61a-9361-4a3b-b3ee-265f924056d0","Type":"ContainerDied","Data":"bcceb4fcce53f9888f6adc1c63eb6595768999b729cd5f864e753510fa7936ae"} Jul 28 19:42:13.890126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.890098 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:13.978518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.978486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle\") pod \"cc70e61a-9361-4a3b-b3ee-265f924056d0\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " Jul 28 19:42:13.978648 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.978585 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util\") pod \"cc70e61a-9361-4a3b-b3ee-265f924056d0\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " Jul 28 19:42:13.978648 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.978615 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcmqj\" (UniqueName: \"kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj\") pod \"cc70e61a-9361-4a3b-b3ee-265f924056d0\" (UID: \"cc70e61a-9361-4a3b-b3ee-265f924056d0\") " Jul 28 19:42:13.979241 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.979217 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle" (OuterVolumeSpecName: "bundle") pod "cc70e61a-9361-4a3b-b3ee-265f924056d0" (UID: "cc70e61a-9361-4a3b-b3ee-265f924056d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:42:13.980688 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.980651 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj" (OuterVolumeSpecName: "kube-api-access-mcmqj") pod "cc70e61a-9361-4a3b-b3ee-265f924056d0" (UID: "cc70e61a-9361-4a3b-b3ee-265f924056d0"). InnerVolumeSpecName "kube-api-access-mcmqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:42:13.984687 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:13.984647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util" (OuterVolumeSpecName: "util") pod "cc70e61a-9361-4a3b-b3ee-265f924056d0" (UID: "cc70e61a-9361-4a3b-b3ee-265f924056d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:42:14.079909 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.079842 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:14.079909 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.079874 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc70e61a-9361-4a3b-b3ee-265f924056d0-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:14.079909 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.079887 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmqj\" (UniqueName: \"kubernetes.io/projected/cc70e61a-9361-4a3b-b3ee-265f924056d0-kube-api-access-mcmqj\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:14.747390 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.747360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" event={"ID":"cc70e61a-9361-4a3b-b3ee-265f924056d0","Type":"ContainerDied","Data":"37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b"} Jul 28 19:42:14.747390 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.747386 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn99jt" Jul 28 19:42:14.747595 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:14.747408 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37aa9f69b2457154a8ee8120a127b5c3a8b05feb9fadceb7d51a2ec2d94c2c3b" Jul 28 19:42:22.057351 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057321 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr"] Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057584 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="pull" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057595 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="pull" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057604 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="extract" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057610 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="extract" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057619 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="util" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057623 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="util" Jul 28 19:42:22.057804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.057686 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc70e61a-9361-4a3b-b3ee-265f924056d0" containerName="extract" Jul 28 19:42:22.064504 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.064484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.067335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.067314 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 19:42:22.067578 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.067563 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-25r4d\"" Jul 28 19:42:22.067855 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.067839 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 19:42:22.067900 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.067882 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 19:42:22.068072 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.068054 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 19:42:22.080722 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.080697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr"] Jul 28 19:42:22.235855 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.235822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.236020 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.235891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spt44\" (UniqueName: \"kubernetes.io/projected/2d55e6cb-7b24-4081-ab58-842287f4e250-kube-api-access-spt44\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.236020 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.235924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.236020 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.235959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.236020 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.235988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.336738 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.336649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.336738 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.336726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spt44\" (UniqueName: \"kubernetes.io/projected/2d55e6cb-7b24-4081-ab58-842287f4e250-kube-api-access-spt44\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.336899 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.336746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.336899 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.336772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.336899 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.336793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.337195 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.337176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.339079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.339058 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d55e6cb-7b24-4081-ab58-842287f4e250-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.339173 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.339156 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.339294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.339275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d55e6cb-7b24-4081-ab58-842287f4e250-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.347033 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.347011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spt44\" (UniqueName: \"kubernetes.io/projected/2d55e6cb-7b24-4081-ab58-842287f4e250-kube-api-access-spt44\") pod \"opendatahub-operator-controller-manager-856746f77b-cpjrr\" (UID: \"2d55e6cb-7b24-4081-ab58-842287f4e250\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.374874 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.374853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:22.532051 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.532025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr"] Jul 28 19:42:22.534315 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:22.534289 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d55e6cb_7b24_4081_ab58_842287f4e250.slice/crio-d672b3564c747408735a8d33f23a1ec32fc517156a55a1c43636917f0826d47a WatchSource:0}: Error finding container d672b3564c747408735a8d33f23a1ec32fc517156a55a1c43636917f0826d47a: Status 404 returned error can't find the container with id d672b3564c747408735a8d33f23a1ec32fc517156a55a1c43636917f0826d47a Jul 28 19:42:22.784213 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:22.784181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" event={"ID":"2d55e6cb-7b24-4081-ab58-842287f4e250","Type":"ContainerStarted","Data":"d672b3564c747408735a8d33f23a1ec32fc517156a55a1c43636917f0826d47a"} Jul 28 19:42:25.799969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:25.799886 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" event={"ID":"2d55e6cb-7b24-4081-ab58-842287f4e250","Type":"ContainerStarted","Data":"4000161777e7e709e93cce851419dfc94a906d7a76b22d98bc5571a8c05d0fc2"} Jul 28 19:42:26.805238 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:26.805141 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d55e6cb-7b24-4081-ab58-842287f4e250" containerID="4000161777e7e709e93cce851419dfc94a906d7a76b22d98bc5571a8c05d0fc2" exitCode=0 Jul 28 19:42:26.805576 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:26.805234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" event={"ID":"2d55e6cb-7b24-4081-ab58-842287f4e250","Type":"ContainerDied","Data":"4000161777e7e709e93cce851419dfc94a906d7a76b22d98bc5571a8c05d0fc2"} Jul 28 19:42:27.811539 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:27.811504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" event={"ID":"2d55e6cb-7b24-4081-ab58-842287f4e250","Type":"ContainerStarted","Data":"f8ad5e85e2f9cc9cf32dd913132e5db479e2511753797c3cee7549677bdcd337"} Jul 28 19:42:27.811956 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:27.811644 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:27.836820 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:27.836774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" podStartSLOduration=1.561529407 podStartE2EDuration="5.836757797s" totalImagesPullingTime="3.247491599s" totalInitContainerRuntime="1.027736791s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:27.83570154 +0000 UTC m=+367.271655493" watchObservedRunningTime="2026-07-28 19:42:27.836757797 +0000 UTC m=+367.272711761" Jul 28 19:42:38.818760 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:38.818732 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-cpjrr" Jul 28 19:42:42.524083 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.524044 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h"] Jul 28 19:42:42.531518 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.531493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.536228 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.536200 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 28 19:42:42.536228 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.536210 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-72bmn\"" Jul 28 19:42:42.536379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.536210 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 28 19:42:42.554996 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.554962 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h"] Jul 28 19:42:42.599811 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.599777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811d12a-4ad1-45ff-87f7-ea72a4225309-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.599990 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.599823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnn4v\" (UniqueName: \"kubernetes.io/projected/2811d12a-4ad1-45ff-87f7-ea72a4225309-kube-api-access-xnn4v\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.599990 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.599887 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2811d12a-4ad1-45ff-87f7-ea72a4225309-tmp\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.701219 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.701115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnn4v\" (UniqueName: \"kubernetes.io/projected/2811d12a-4ad1-45ff-87f7-ea72a4225309-kube-api-access-xnn4v\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.701219 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.701185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2811d12a-4ad1-45ff-87f7-ea72a4225309-tmp\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.701458 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.701253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811d12a-4ad1-45ff-87f7-ea72a4225309-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.703706 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.703676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2811d12a-4ad1-45ff-87f7-ea72a4225309-tmp\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.703951 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.703932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811d12a-4ad1-45ff-87f7-ea72a4225309-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.740117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.740080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnn4v\" (UniqueName: \"kubernetes.io/projected/2811d12a-4ad1-45ff-87f7-ea72a4225309-kube-api-access-xnn4v\") pod \"kube-auth-proxy-77597fcbdb-vsl7h\" (UID: \"2811d12a-4ad1-45ff-87f7-ea72a4225309\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:42.841057 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:42.840980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" Jul 28 19:42:43.038462 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:43.038422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h"] Jul 28 19:42:43.041076 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:43.041049 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2811d12a_4ad1_45ff_87f7_ea72a4225309.slice/crio-ff8590a30564730753973a7ca2c60f3c5a1f0de3ce0c1e186991b8f043c3002a WatchSource:0}: Error finding container ff8590a30564730753973a7ca2c60f3c5a1f0de3ce0c1e186991b8f043c3002a: Status 404 returned error can't find the container with id ff8590a30564730753973a7ca2c60f3c5a1f0de3ce0c1e186991b8f043c3002a Jul 28 19:42:43.887240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:43.887201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" event={"ID":"2811d12a-4ad1-45ff-87f7-ea72a4225309","Type":"ContainerStarted","Data":"ff8590a30564730753973a7ca2c60f3c5a1f0de3ce0c1e186991b8f043c3002a"} Jul 28 19:42:44.113116 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.113086 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5ktdt"] Jul 28 19:42:44.120293 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.120259 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.123379 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.123232 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jlspn\"" Jul 28 19:42:44.123833 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.123807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 28 19:42:44.129552 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.129529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5ktdt"] Jul 28 19:42:44.214579 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.214547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z9w\" (UniqueName: \"kubernetes.io/projected/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-kube-api-access-k7z9w\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.214758 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.214614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-tls-certs\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.315889 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.315857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-tls-certs\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.316069 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.315928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z9w\" (UniqueName: \"kubernetes.io/projected/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-kube-api-access-k7z9w\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.318654 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.318624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-tls-certs\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.344142 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.344112 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z9w\" (UniqueName: \"kubernetes.io/projected/e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7-kube-api-access-k7z9w\") pod \"model-serving-api-54d89bd5fc-5ktdt\" (UID: \"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7\") " pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.434290 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.434257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:44.572647 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.572601 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:44.579280 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.579226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.582193 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.582168 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 19:42:44.582347 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.582326 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 19:42:44.582462 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.582381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bmgxf\"" Jul 28 19:42:44.582529 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.582289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 19:42:44.582711 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.582620 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 19:42:44.595639 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.595617 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:44.609997 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.609976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5ktdt"] Jul 28 19:42:44.618202 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618304 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618358 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d8cq\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618401 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618443 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618509 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.618589 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.618549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719609 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d8cq\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.719852 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.719849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.720819 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.720787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.722465 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.722427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.722635 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.722610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.723093 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.723066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.723305 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.723277 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.728385 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.728342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d8cq\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.728555 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.728536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9scz5\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:44.891513 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:44.891374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:45.246311 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:45.246264 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a1c34a_b4fb_4866_bf3b_c8f7f3d514a7.slice/crio-efd296dc58efb5d261a1f70c7307912663710b2ef1a7a411adbc277a0c75931d WatchSource:0}: Error finding container efd296dc58efb5d261a1f70c7307912663710b2ef1a7a411adbc277a0c75931d: Status 404 returned error can't find the container with id efd296dc58efb5d261a1f70c7307912663710b2ef1a7a411adbc277a0c75931d Jul 28 19:42:45.388906 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:45.388638 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:45.391973 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:45.391944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab96236e_9b31_49b9_84b7_2d2b5bae6e82.slice/crio-2d12fc6bf5b35809d5ad6d66737a33db45da875730dfe1d21df2b0be4ee0fc58 WatchSource:0}: Error finding container 2d12fc6bf5b35809d5ad6d66737a33db45da875730dfe1d21df2b0be4ee0fc58: Status 404 returned error can't find the container with id 2d12fc6bf5b35809d5ad6d66737a33db45da875730dfe1d21df2b0be4ee0fc58 Jul 28 19:42:45.901763 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:45.901717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" event={"ID":"2811d12a-4ad1-45ff-87f7-ea72a4225309","Type":"ContainerStarted","Data":"e31aeb3ea6cbee570ad8279f46aedf7d14203105566e1c86f67d8ab27bafc477"} Jul 28 19:42:45.903233 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:45.903201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" event={"ID":"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7","Type":"ContainerStarted","Data":"efd296dc58efb5d261a1f70c7307912663710b2ef1a7a411adbc277a0c75931d"} Jul 28 19:42:45.904499 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:45.904476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" event={"ID":"ab96236e-9b31-49b9-84b7-2d2b5bae6e82","Type":"ContainerStarted","Data":"2d12fc6bf5b35809d5ad6d66737a33db45da875730dfe1d21df2b0be4ee0fc58"} Jul 28 19:42:45.921618 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:45.921573 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-vsl7h" podStartSLOduration=1.653428372 podStartE2EDuration="3.921558671s" totalImagesPullingTime="2.268130299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:45.919594949 +0000 UTC m=+385.355548902" watchObservedRunningTime="2026-07-28 19:42:45.921558671 +0000 UTC m=+385.357512621" Jul 28 19:42:47.917213 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:47.917178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" event={"ID":"e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7","Type":"ContainerStarted","Data":"509d1f7a972a2ecf388bd622756f69e45c3584384bfb2d7436a3dbf913fcfddd"} Jul 28 19:42:47.917620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:47.917261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:42:47.939099 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:47.939047 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" podStartSLOduration=1.987190112 podStartE2EDuration="3.9390319s" totalImagesPullingTime="1.951841788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:47.937410789 +0000 UTC m=+387.373364755" watchObservedRunningTime="2026-07-28 19:42:47.9390319 +0000 UTC m=+387.374985892" Jul 28 19:42:48.240649 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:48.240611 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:42:48.240772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:48.240697 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:42:48.929541 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:48.929505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" event={"ID":"ab96236e-9b31-49b9-84b7-2d2b5bae6e82","Type":"ContainerStarted","Data":"db27e063ec8bdf78f181fc5457c2956f778dc11eddb3793fa5630178b3f65967"} Jul 28 19:42:48.929985 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:48.929598 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:48.954794 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:48.954735 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" podStartSLOduration=2.108768708 podStartE2EDuration="4.95471481s" totalImagesPullingTime="2.845946102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:48.952291645 +0000 UTC m=+388.388245601" watchObservedRunningTime="2026-07-28 19:42:48.95471481 +0000 UTC m=+388.390668763" Jul 28 19:42:49.935532 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:49.935500 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9scz5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 19:42:49.935919 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:49.935555 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 19:42:52.934813 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:52.934782 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:54.026057 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.026022 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dw87p"] Jul 28 19:42:54.029350 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.029333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.048181 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.048153 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l8cb8\"" Jul 28 19:42:54.048302 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.048215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 28 19:42:54.051426 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.051401 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq"] Jul 28 19:42:54.054515 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.054495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.060911 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.060748 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nllhh\"" Jul 28 19:42:54.071064 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.071042 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dw87p"] Jul 28 19:42:54.097667 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.097635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvpn\" (UniqueName: \"kubernetes.io/projected/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-kube-api-access-dkvpn\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.097770 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.097700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.097770 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.097733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6c7\" (UniqueName: \"kubernetes.io/projected/5bef0052-9412-40b4-a364-01ed66d9ac6f-kube-api-access-jg6c7\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.097855 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.097818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bef0052-9412-40b4-a364-01ed66d9ac6f-cert\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.101587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.101572 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 28 19:42:54.127316 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.127292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq"] Jul 28 19:42:54.198283 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.198256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bef0052-9412-40b4-a364-01ed66d9ac6f-cert\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.198414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.198310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvpn\" (UniqueName: \"kubernetes.io/projected/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-kube-api-access-dkvpn\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.198414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.198333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.198414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.198349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6c7\" (UniqueName: \"kubernetes.io/projected/5bef0052-9412-40b4-a364-01ed66d9ac6f-kube-api-access-jg6c7\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.198526 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:42:54.198482 2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 28 19:42:54.198567 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:42:54.198541 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert podName:69d9ce3b-6cd9-4051-83b6-cb5c92da7d05 nodeName:}" failed. No retries permitted until 2026-07-28 19:42:54.698522219 +0000 UTC m=+394.134476149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert") pod "llmisvc-controller-manager-777c6bd7d5-qkhvq" (UID: "69d9ce3b-6cd9-4051-83b6-cb5c92da7d05") : secret "llmisvc-webhook-server-cert" not found Jul 28 19:42:54.200846 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.200827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bef0052-9412-40b4-a364-01ed66d9ac6f-cert\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.218234 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.218204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvpn\" (UniqueName: \"kubernetes.io/projected/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-kube-api-access-dkvpn\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.224719 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.224699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6c7\" (UniqueName: \"kubernetes.io/projected/5bef0052-9412-40b4-a364-01ed66d9ac6f-kube-api-access-jg6c7\") pod \"kserve-controller-manager-d6f4b65b8-dw87p\" (UID: \"5bef0052-9412-40b4-a364-01ed66d9ac6f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.340054 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.339985 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:54.516126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.516104 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dw87p"] Jul 28 19:42:54.518394 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:54.518370 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bef0052_9412_40b4_a364_01ed66d9ac6f.slice/crio-d0b9c9e2353c6bf24c992000accfcaef6aaa70c9baa531ba09829a48340ae2af WatchSource:0}: Error finding container d0b9c9e2353c6bf24c992000accfcaef6aaa70c9baa531ba09829a48340ae2af: Status 404 returned error can't find the container with id d0b9c9e2353c6bf24c992000accfcaef6aaa70c9baa531ba09829a48340ae2af Jul 28 19:42:54.701880 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.701846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.704180 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.704161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d9ce3b-6cd9-4051-83b6-cb5c92da7d05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qkhvq\" (UID: \"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:54.958035 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.957961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" event={"ID":"5bef0052-9412-40b4-a364-01ed66d9ac6f","Type":"ContainerStarted","Data":"d0b9c9e2353c6bf24c992000accfcaef6aaa70c9baa531ba09829a48340ae2af"} Jul 28 19:42:54.964155 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:54.964135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:55.112712 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.112679 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq"] Jul 28 19:42:55.117565 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:42:55.117538 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69d9ce3b_6cd9_4051_83b6_cb5c92da7d05.slice/crio-6a803e90f199f5b24a8a96ad8d9060745853b55edbebbe5fdd50fcf137833c7c WatchSource:0}: Error finding container 6a803e90f199f5b24a8a96ad8d9060745853b55edbebbe5fdd50fcf137833c7c: Status 404 returned error can't find the container with id 6a803e90f199f5b24a8a96ad8d9060745853b55edbebbe5fdd50fcf137833c7c Jul 28 19:42:55.807813 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.807780 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:55.808021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.807976 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" containerID="cri-o://db27e063ec8bdf78f181fc5457c2956f778dc11eddb3793fa5630178b3f65967" gracePeriod=30 Jul 28 19:42:55.934723 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.934694 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9scz5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:8080/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" start-of-body= Jul 28 19:42:55.934875 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.934744 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.31:8080/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" Jul 28 19:42:55.964521 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.964488 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerID="db27e063ec8bdf78f181fc5457c2956f778dc11eddb3793fa5630178b3f65967" exitCode=0 Jul 28 19:42:55.964680 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.964553 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" event={"ID":"ab96236e-9b31-49b9-84b7-2d2b5bae6e82","Type":"ContainerDied","Data":"db27e063ec8bdf78f181fc5457c2956f778dc11eddb3793fa5630178b3f65967"} Jul 28 19:42:55.965787 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:55.965765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" event={"ID":"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05","Type":"ContainerStarted","Data":"6a803e90f199f5b24a8a96ad8d9060745853b55edbebbe5fdd50fcf137833c7c"} Jul 28 19:42:56.228164 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.227358 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:56.317127 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317162 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317200 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d8cq\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317236 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317323 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317306 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317549 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.317549 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.317386 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap\") pod \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\" (UID: \"ab96236e-9b31-49b9-84b7-2d2b5bae6e82\") " Jul 28 19:42:56.318801 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.318046 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 19:42:56.321835 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.321779 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq" (OuterVolumeSpecName: "kube-api-access-9d8cq") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "kube-api-access-9d8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:42:56.322298 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.322257 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:42:56.327223 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.327186 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs" (OuterVolumeSpecName: "local-certs") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:42:56.332267 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.332227 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:42:56.333562 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.332949 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:42:56.334116 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.334092 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts" (OuterVolumeSpecName: "cacerts") pod "ab96236e-9b31-49b9-84b7-2d2b5bae6e82" (UID: "ab96236e-9b31-49b9-84b7-2d2b5bae6e82"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418387 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-dns-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418429 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d8cq\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-kube-api-access-9d8cq\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418446 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418460 2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-local-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418472 2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-cacerts\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418485 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-csr-ca-configmap\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.418977 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.418500 2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab96236e-9b31-49b9-84b7-2d2b5bae6e82-istio-kubeconfig\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:42:56.972632 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.972605 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" Jul 28 19:42:56.972793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.972632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5" event={"ID":"ab96236e-9b31-49b9-84b7-2d2b5bae6e82","Type":"ContainerDied","Data":"2d12fc6bf5b35809d5ad6d66737a33db45da875730dfe1d21df2b0be4ee0fc58"} Jul 28 19:42:56.972793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:56.972701 2562 scope.go:117] "RemoveContainer" containerID="db27e063ec8bdf78f181fc5457c2956f778dc11eddb3793fa5630178b3f65967" Jul 28 19:42:57.015867 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.015835 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:57.028150 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.028122 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9scz5"] Jul 28 19:42:57.163651 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.163624 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" path="/var/lib/kubelet/pods/ab96236e-9b31-49b9-84b7-2d2b5bae6e82/volumes" Jul 28 19:42:57.979145 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.979095 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" event={"ID":"5bef0052-9412-40b4-a364-01ed66d9ac6f","Type":"ContainerStarted","Data":"94bfa6d9501ce63ba5c524c0ba47a865b9cd2060b5a48d637e31f8f5eca6a98b"} Jul 28 19:42:57.979585 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.979282 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:42:57.980593 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.980572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" event={"ID":"69d9ce3b-6cd9-4051-83b6-cb5c92da7d05","Type":"ContainerStarted","Data":"a2d998f9183019f618e466733730634cb960dfbf43f2a680636185b5a0093c99"} Jul 28 19:42:57.980710 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.980629 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:42:57.999015 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:57.998932 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" podStartSLOduration=2.323450302 podStartE2EDuration="4.998918957s" totalImagesPullingTime="2.675468655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:57.997940495 +0000 UTC m=+397.433894462" watchObservedRunningTime="2026-07-28 19:42:57.998918957 +0000 UTC m=+397.434872937" Jul 28 19:42:58.019527 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:58.019482 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" podStartSLOduration=1.3480020449999999 podStartE2EDuration="4.019468132s" totalImagesPullingTime="2.671466087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:58.016655901 +0000 UTC m=+397.452609853" watchObservedRunningTime="2026-07-28 19:42:58.019468132 +0000 UTC m=+397.455422088" Jul 28 19:42:58.933324 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:42:58.933292 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5ktdt" Jul 28 19:43:28.987651 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:28.987620 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qkhvq" Jul 28 19:43:28.990731 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:28.990711 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dw87p" Jul 28 19:43:43.195408 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.195375 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:43.195802 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.195736 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" Jul 28 19:43:43.195802 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.195752 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" Jul 28 19:43:43.195877 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.195810 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab96236e-9b31-49b9-84b7-2d2b5bae6e82" containerName="discovery" Jul 28 19:43:43.202913 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.202884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:43.205961 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.205932 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 19:43:43.206593 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.206254 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6gpms\"" Jul 28 19:43:43.206593 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.206420 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 19:43:43.208762 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.208736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:43.306761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.306730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl6l9\" (UniqueName: \"kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9\") pod \"kuadrant-operator-catalog-tnpn5\" (UID: \"7ae649d4-ce62-498b-a6a7-ea49a843e9e2\") " pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:43.407906 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.407871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6l9\" (UniqueName: \"kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9\") pod \"kuadrant-operator-catalog-tnpn5\" (UID: \"7ae649d4-ce62-498b-a6a7-ea49a843e9e2\") " pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:43.416300 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.416268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6l9\" (UniqueName: \"kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9\") pod \"kuadrant-operator-catalog-tnpn5\" (UID: \"7ae649d4-ce62-498b-a6a7-ea49a843e9e2\") " pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:43.516638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.516518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:43.569746 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.569709 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:43.651749 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.651724 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:43.654583 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:43:43.654558 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae649d4_ce62_498b_a6a7_ea49a843e9e2.slice/crio-fff0b1a2339e459679fe88db46e16ec5d3d5d1bb0641cae31b3092dc0d583082 WatchSource:0}: Error finding container fff0b1a2339e459679fe88db46e16ec5d3d5d1bb0641cae31b3092dc0d583082: Status 404 returned error can't find the container with id fff0b1a2339e459679fe88db46e16ec5d3d5d1bb0641cae31b3092dc0d583082 Jul 28 19:43:43.775110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.775036 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ld4"] Jul 28 19:43:43.777927 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.777905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:43.786603 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.786581 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ld4"] Jul 28 19:43:43.810394 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.810363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh5n\" (UniqueName: \"kubernetes.io/projected/66f98abf-d517-4cf7-a99f-3e811610ed9c-kube-api-access-nhh5n\") pod \"kuadrant-operator-catalog-62ld4\" (UID: \"66f98abf-d517-4cf7-a99f-3e811610ed9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:43.911490 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.911449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh5n\" (UniqueName: \"kubernetes.io/projected/66f98abf-d517-4cf7-a99f-3e811610ed9c-kube-api-access-nhh5n\") pod \"kuadrant-operator-catalog-62ld4\" (UID: \"66f98abf-d517-4cf7-a99f-3e811610ed9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:43.919515 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:43.919490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh5n\" (UniqueName: \"kubernetes.io/projected/66f98abf-d517-4cf7-a99f-3e811610ed9c-kube-api-access-nhh5n\") pod \"kuadrant-operator-catalog-62ld4\" (UID: \"66f98abf-d517-4cf7-a99f-3e811610ed9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:44.088128 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:44.088043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:44.195055 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:44.195015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" event={"ID":"7ae649d4-ce62-498b-a6a7-ea49a843e9e2","Type":"ContainerStarted","Data":"fff0b1a2339e459679fe88db46e16ec5d3d5d1bb0641cae31b3092dc0d583082"} Jul 28 19:43:44.237539 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:44.237506 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ld4"] Jul 28 19:43:44.240045 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:43:44.240000 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f98abf_d517_4cf7_a99f_3e811610ed9c.slice/crio-26d07b7423c3e00e3ac7eaa835f96ac19ed7d438fbc5a45ff5043d4edc9f96d5 WatchSource:0}: Error finding container 26d07b7423c3e00e3ac7eaa835f96ac19ed7d438fbc5a45ff5043d4edc9f96d5: Status 404 returned error can't find the container with id 26d07b7423c3e00e3ac7eaa835f96ac19ed7d438fbc5a45ff5043d4edc9f96d5 Jul 28 19:43:45.204578 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:45.204539 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" event={"ID":"66f98abf-d517-4cf7-a99f-3e811610ed9c","Type":"ContainerStarted","Data":"26d07b7423c3e00e3ac7eaa835f96ac19ed7d438fbc5a45ff5043d4edc9f96d5"} Jul 28 19:43:46.213158 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.213122 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" event={"ID":"7ae649d4-ce62-498b-a6a7-ea49a843e9e2","Type":"ContainerStarted","Data":"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8"} Jul 28 19:43:46.213649 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.213192 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" podUID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" containerName="registry-server" containerID="cri-o://d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8" gracePeriod=2 Jul 28 19:43:46.214797 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.214769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" event={"ID":"66f98abf-d517-4cf7-a99f-3e811610ed9c","Type":"ContainerStarted","Data":"48ed6f3837c22cf92b22cf5377887d4812884f39ac6e2a44a9f617a952e7f05b"} Jul 28 19:43:46.230246 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.230209 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" podStartSLOduration=1.171751338 podStartE2EDuration="3.230195068s" totalImagesPullingTime="2.05844373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:43:46.228404229 +0000 UTC m=+445.664358182" watchObservedRunningTime="2026-07-28 19:43:46.230195068 +0000 UTC m=+445.666149019" Jul 28 19:43:46.247802 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.247765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" podStartSLOduration=1.7741050299999999 podStartE2EDuration="3.247752946s" totalImagesPullingTime="1.473647916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:43:46.247429491 +0000 UTC m=+445.683383442" watchObservedRunningTime="2026-07-28 19:43:46.247752946 +0000 UTC m=+445.683706897" Jul 28 19:43:46.455060 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.455037 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:46.533706 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.533630 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl6l9\" (UniqueName: \"kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9\") pod \"7ae649d4-ce62-498b-a6a7-ea49a843e9e2\" (UID: \"7ae649d4-ce62-498b-a6a7-ea49a843e9e2\") " Jul 28 19:43:46.535625 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.535603 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9" (OuterVolumeSpecName: "kube-api-access-gl6l9") pod "7ae649d4-ce62-498b-a6a7-ea49a843e9e2" (UID: "7ae649d4-ce62-498b-a6a7-ea49a843e9e2"). InnerVolumeSpecName "kube-api-access-gl6l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:43:46.634615 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:46.634587 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl6l9\" (UniqueName: \"kubernetes.io/projected/7ae649d4-ce62-498b-a6a7-ea49a843e9e2-kube-api-access-gl6l9\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:43:47.221334 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.221303 2562 generic.go:356] "Generic (PLEG): container finished" podID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" containerID="d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8" exitCode=0 Jul 28 19:43:47.221716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.221366 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" Jul 28 19:43:47.221716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.221372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" event={"ID":"7ae649d4-ce62-498b-a6a7-ea49a843e9e2","Type":"ContainerDied","Data":"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8"} Jul 28 19:43:47.221716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.221430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tnpn5" event={"ID":"7ae649d4-ce62-498b-a6a7-ea49a843e9e2","Type":"ContainerDied","Data":"fff0b1a2339e459679fe88db46e16ec5d3d5d1bb0641cae31b3092dc0d583082"} Jul 28 19:43:47.221716 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.221457 2562 scope.go:117] "RemoveContainer" containerID="d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8" Jul 28 19:43:47.230311 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.230292 2562 scope.go:117] "RemoveContainer" containerID="d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8" Jul 28 19:43:47.230569 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:43:47.230550 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8\": container with ID starting with d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8 not found: ID does not exist" containerID="d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8" Jul 28 19:43:47.230632 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.230579 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8"} err="failed to get container status \"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8\": rpc error: code = NotFound desc = could not find container \"d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8\": container with ID starting with d4661f7ebee35f6583a218f8c434e903b2c7a7ca986a5be46d09f33718b0c0d8 not found: ID does not exist" Jul 28 19:43:47.238421 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.238399 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:47.242300 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:47.242280 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tnpn5"] Jul 28 19:43:49.161514 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:49.161484 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" path="/var/lib/kubelet/pods/7ae649d4-ce62-498b-a6a7-ea49a843e9e2/volumes" Jul 28 19:43:54.088865 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:54.088833 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:54.089288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:54.088894 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:54.110432 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:54.110407 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:43:54.274619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:43:54.274584 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-62ld4" Jul 28 19:44:02.980924 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.980896 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw"] Jul 28 19:44:02.981338 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.981209 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" containerName="registry-server" Jul 28 19:44:02.981338 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.981219 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" containerName="registry-server" Jul 28 19:44:02.981338 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.981286 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae649d4-ce62-498b-a6a7-ea49a843e9e2" containerName="registry-server" Jul 28 19:44:02.985575 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.985556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:02.988057 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.988038 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q24p6\"" Jul 28 19:44:02.991962 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:02.991940 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw"] Jul 28 19:44:03.064730 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.064699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9kd\" (UniqueName: \"kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.064859 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.064740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.064859 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.064778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.078940 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.078911 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr"] Jul 28 19:44:03.082249 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.082231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.089793 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.089770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr"] Jul 28 19:44:03.165234 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9kd\" (UniqueName: \"kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.165397 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.165397 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.165826 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.170881 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.170993 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.165962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.170993 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.170900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.170993 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.170962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6dw\" (UniqueName: \"kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.173829 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.173810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9kd\" (UniqueName: \"kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.271630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.271554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.271630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.271601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6dw\" (UniqueName: \"kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.271830 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.271656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.271996 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.271976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.272063 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.271986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.279587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.279559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6dw\" (UniqueName: \"kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.296159 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.296139 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:03.380331 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.380298 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz"] Jul 28 19:44:03.385755 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.385734 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.391050 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.391028 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz"] Jul 28 19:44:03.391198 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.391176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:03.431961 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.431928 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw"] Jul 28 19:44:03.434098 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:03.434073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53aad1db_8cad_44da_be15_9d65419096bb.slice/crio-dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28 WatchSource:0}: Error finding container dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28: Status 404 returned error can't find the container with id dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28 Jul 28 19:44:03.473785 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.473754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.473893 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.473815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvjk\" (UniqueName: \"kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.473893 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.473839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.535398 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.535370 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr"] Jul 28 19:44:03.538012 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:03.537984 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415a9325_f24b_4704_9204_1ff2cb43d51d.slice/crio-79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa WatchSource:0}: Error finding container 79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa: Status 404 returned error can't find the container with id 79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa Jul 28 19:44:03.575039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.575013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.575126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.575046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvjk\" (UniqueName: \"kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.575126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.575064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.575335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.575319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.575377 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.575356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.582806 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.582785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvjk\" (UniqueName: \"kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.699918 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.699891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:03.780969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.780939 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj"] Jul 28 19:44:03.786287 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.786228 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.793588 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.793559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj"] Jul 28 19:44:03.833553 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.833531 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz"] Jul 28 19:44:03.835443 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:03.835419 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36cc7cf_d61a_4270_9199_baeb90d78dbe.slice/crio-7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb WatchSource:0}: Error finding container 7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb: Status 404 returned error can't find the container with id 7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb Jul 28 19:44:03.877685 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.877645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.877771 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.877718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.877771 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.877766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhl2g\" (UniqueName: \"kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.978998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.978971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.979118 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.979010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.979118 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.979068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhl2g\" (UniqueName: \"kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.979391 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.979371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.979434 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.979377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:03.988109 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:03.988088 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhl2g\" (UniqueName: \"kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:04.097529 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.097466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:04.228828 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.228802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj"] Jul 28 19:44:04.230796 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:04.230769 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271d3d76_5efa_4337_9fe9_def48a6ee259.slice/crio-77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8 WatchSource:0}: Error finding container 77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8: Status 404 returned error can't find the container with id 77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8 Jul 28 19:44:04.301812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.301788 2562 generic.go:356] "Generic (PLEG): container finished" podID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerID="bfef113029ebc537e5a103f1bd4f46fb4b510a5ad5392b8ff20df6bf79a282cb" exitCode=0 Jul 28 19:44:04.301912 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.301871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" event={"ID":"f36cc7cf-d61a-4270-9199-baeb90d78dbe","Type":"ContainerDied","Data":"bfef113029ebc537e5a103f1bd4f46fb4b510a5ad5392b8ff20df6bf79a282cb"} Jul 28 19:44:04.301969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.301925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" event={"ID":"f36cc7cf-d61a-4270-9199-baeb90d78dbe","Type":"ContainerStarted","Data":"7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb"} Jul 28 19:44:04.303548 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.303527 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerStarted","Data":"674e66ce988e8e47ec5af73b7e2bc800973dc7940eb344be662e3cf09dee8735"} Jul 28 19:44:04.303680 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.303569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerStarted","Data":"77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8"} Jul 28 19:44:04.305217 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.305194 2562 generic.go:356] "Generic (PLEG): container finished" podID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerID="3bf92dcc1b8994472de091baebf8b0d0de7c425399ff5d81e41dc3b8ff7a74b4" exitCode=0 Jul 28 19:44:04.305300 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.305274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" event={"ID":"415a9325-f24b-4704-9204-1ff2cb43d51d","Type":"ContainerDied","Data":"3bf92dcc1b8994472de091baebf8b0d0de7c425399ff5d81e41dc3b8ff7a74b4"} Jul 28 19:44:04.305360 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.305316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" event={"ID":"415a9325-f24b-4704-9204-1ff2cb43d51d","Type":"ContainerStarted","Data":"79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa"} Jul 28 19:44:04.307010 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.306990 2562 generic.go:356] "Generic (PLEG): container finished" podID="53aad1db-8cad-44da-be15-9d65419096bb" containerID="017b1c42922b73a3e8335514ca43e2276c73cb4e48b9ee943e89c83a1f0bc451" exitCode=0 Jul 28 19:44:04.307102 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.307074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" event={"ID":"53aad1db-8cad-44da-be15-9d65419096bb","Type":"ContainerDied","Data":"017b1c42922b73a3e8335514ca43e2276c73cb4e48b9ee943e89c83a1f0bc451"} Jul 28 19:44:04.307153 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:04.307123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" event={"ID":"53aad1db-8cad-44da-be15-9d65419096bb","Type":"ContainerStarted","Data":"dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28"} Jul 28 19:44:05.313755 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.313725 2562 generic.go:356] "Generic (PLEG): container finished" podID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerID="674e66ce988e8e47ec5af73b7e2bc800973dc7940eb344be662e3cf09dee8735" exitCode=0 Jul 28 19:44:05.314072 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.313806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerDied","Data":"674e66ce988e8e47ec5af73b7e2bc800973dc7940eb344be662e3cf09dee8735"} Jul 28 19:44:05.315721 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.315702 2562 generic.go:356] "Generic (PLEG): container finished" podID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerID="e20ed599e9f3b2ee837775829c9675ad4a1f13da7780f878ef64483acfdc9b68" exitCode=0 Jul 28 19:44:05.315834 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.315772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" event={"ID":"415a9325-f24b-4704-9204-1ff2cb43d51d","Type":"ContainerDied","Data":"e20ed599e9f3b2ee837775829c9675ad4a1f13da7780f878ef64483acfdc9b68"} Jul 28 19:44:05.317808 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.317783 2562 generic.go:356] "Generic (PLEG): container finished" podID="53aad1db-8cad-44da-be15-9d65419096bb" containerID="2fd26cc33e619a8c09a739f8dbee0102ba60888fc43c44f1af09b516646cdea2" exitCode=0 Jul 28 19:44:05.317997 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.317812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" event={"ID":"53aad1db-8cad-44da-be15-9d65419096bb","Type":"ContainerDied","Data":"2fd26cc33e619a8c09a739f8dbee0102ba60888fc43c44f1af09b516646cdea2"} Jul 28 19:44:05.320072 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.320049 2562 generic.go:356] "Generic (PLEG): container finished" podID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerID="052b20d9b76be626e9d2ad9088a39a295265b724b0f75a41aa7616c7d6eaab79" exitCode=0 Jul 28 19:44:05.320180 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:05.320090 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" event={"ID":"f36cc7cf-d61a-4270-9199-baeb90d78dbe","Type":"ContainerDied","Data":"052b20d9b76be626e9d2ad9088a39a295265b724b0f75a41aa7616c7d6eaab79"} Jul 28 19:44:06.327085 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.327056 2562 generic.go:356] "Generic (PLEG): container finished" podID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerID="c994f757177afc8f4af3373e2ac5c1c54ee29ae5adbaaf70eb439605745eed05" exitCode=0 Jul 28 19:44:06.327417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.327143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" event={"ID":"415a9325-f24b-4704-9204-1ff2cb43d51d","Type":"ContainerDied","Data":"c994f757177afc8f4af3373e2ac5c1c54ee29ae5adbaaf70eb439605745eed05"} Jul 28 19:44:06.329168 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.329146 2562 generic.go:356] "Generic (PLEG): container finished" podID="53aad1db-8cad-44da-be15-9d65419096bb" containerID="ffce6a715585b8d75d3980a2e63132ff9a1abb5e71f344faa8ab9b0204d61511" exitCode=0 Jul 28 19:44:06.329265 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.329207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" event={"ID":"53aad1db-8cad-44da-be15-9d65419096bb","Type":"ContainerDied","Data":"ffce6a715585b8d75d3980a2e63132ff9a1abb5e71f344faa8ab9b0204d61511"} Jul 28 19:44:06.331380 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.331359 2562 generic.go:356] "Generic (PLEG): container finished" podID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerID="31351342b4128e4e10b02bcfc8c428d86b46d067224b821bbfe9dd2f0e6311f0" exitCode=0 Jul 28 19:44:06.331481 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.331439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" event={"ID":"f36cc7cf-d61a-4270-9199-baeb90d78dbe","Type":"ContainerDied","Data":"31351342b4128e4e10b02bcfc8c428d86b46d067224b821bbfe9dd2f0e6311f0"} Jul 28 19:44:06.333094 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.333078 2562 generic.go:356] "Generic (PLEG): container finished" podID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerID="68f73d0d7da5493fd9631bb68e0fb748e01eb709050ccb768a829d8eb939cb98" exitCode=0 Jul 28 19:44:06.333175 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:06.333108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerDied","Data":"68f73d0d7da5493fd9631bb68e0fb748e01eb709050ccb768a829d8eb939cb98"} Jul 28 19:44:07.339414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.339384 2562 generic.go:356] "Generic (PLEG): container finished" podID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerID="21a4540e8b243063d87b0ebf50d918222f580b7dc03aaa1cedc9f9285e792387" exitCode=0 Jul 28 19:44:07.339804 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.339468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerDied","Data":"21a4540e8b243063d87b0ebf50d918222f580b7dc03aaa1cedc9f9285e792387"} Jul 28 19:44:07.485712 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.485689 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:07.534035 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.534013 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:07.537849 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.537830 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:07.613092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613028 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk6dw\" (UniqueName: \"kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw\") pod \"415a9325-f24b-4704-9204-1ff2cb43d51d\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " Jul 28 19:44:07.613092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613078 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util\") pod \"53aad1db-8cad-44da-be15-9d65419096bb\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " Jul 28 19:44:07.613274 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util\") pod \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " Jul 28 19:44:07.613274 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util\") pod \"415a9325-f24b-4704-9204-1ff2cb43d51d\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " Jul 28 19:44:07.613274 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613211 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle\") pod \"53aad1db-8cad-44da-be15-9d65419096bb\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " Jul 28 19:44:07.613433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613283 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w9kd\" (UniqueName: \"kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd\") pod \"53aad1db-8cad-44da-be15-9d65419096bb\" (UID: \"53aad1db-8cad-44da-be15-9d65419096bb\") " Jul 28 19:44:07.613433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613309 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle\") pod \"415a9325-f24b-4704-9204-1ff2cb43d51d\" (UID: \"415a9325-f24b-4704-9204-1ff2cb43d51d\") " Jul 28 19:44:07.613433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613342 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle\") pod \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " Jul 28 19:44:07.613433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.613372 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvjk\" (UniqueName: \"kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk\") pod \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\" (UID: \"f36cc7cf-d61a-4270-9199-baeb90d78dbe\") " Jul 28 19:44:07.614147 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.614089 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle" (OuterVolumeSpecName: "bundle") pod "53aad1db-8cad-44da-be15-9d65419096bb" (UID: "53aad1db-8cad-44da-be15-9d65419096bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.614253 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.614170 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle" (OuterVolumeSpecName: "bundle") pod "415a9325-f24b-4704-9204-1ff2cb43d51d" (UID: "415a9325-f24b-4704-9204-1ff2cb43d51d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.614636 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.614612 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle" (OuterVolumeSpecName: "bundle") pod "f36cc7cf-d61a-4270-9199-baeb90d78dbe" (UID: "f36cc7cf-d61a-4270-9199-baeb90d78dbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.615476 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.615446 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw" (OuterVolumeSpecName: "kube-api-access-nk6dw") pod "415a9325-f24b-4704-9204-1ff2cb43d51d" (UID: "415a9325-f24b-4704-9204-1ff2cb43d51d"). InnerVolumeSpecName "kube-api-access-nk6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:07.616013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.615988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk" (OuterVolumeSpecName: "kube-api-access-fhvjk") pod "f36cc7cf-d61a-4270-9199-baeb90d78dbe" (UID: "f36cc7cf-d61a-4270-9199-baeb90d78dbe"). InnerVolumeSpecName "kube-api-access-fhvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:07.616084 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.616021 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd" (OuterVolumeSpecName: "kube-api-access-4w9kd") pod "53aad1db-8cad-44da-be15-9d65419096bb" (UID: "53aad1db-8cad-44da-be15-9d65419096bb"). InnerVolumeSpecName "kube-api-access-4w9kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:07.619687 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.619647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util" (OuterVolumeSpecName: "util") pod "f36cc7cf-d61a-4270-9199-baeb90d78dbe" (UID: "f36cc7cf-d61a-4270-9199-baeb90d78dbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.621180 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.621158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util" (OuterVolumeSpecName: "util") pod "415a9325-f24b-4704-9204-1ff2cb43d51d" (UID: "415a9325-f24b-4704-9204-1ff2cb43d51d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.621953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.621934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util" (OuterVolumeSpecName: "util") pod "53aad1db-8cad-44da-be15-9d65419096bb" (UID: "53aad1db-8cad-44da-be15-9d65419096bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:07.714548 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714525 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714548 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714544 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714548 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714553 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714561 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9kd\" (UniqueName: \"kubernetes.io/projected/53aad1db-8cad-44da-be15-9d65419096bb-kube-api-access-4w9kd\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714570 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/415a9325-f24b-4704-9204-1ff2cb43d51d-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714577 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f36cc7cf-d61a-4270-9199-baeb90d78dbe-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714584 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvjk\" (UniqueName: \"kubernetes.io/projected/f36cc7cf-d61a-4270-9199-baeb90d78dbe-kube-api-access-fhvjk\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714592 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6dw\" (UniqueName: \"kubernetes.io/projected/415a9325-f24b-4704-9204-1ff2cb43d51d-kube-api-access-nk6dw\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:07.714732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:07.714600 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53aad1db-8cad-44da-be15-9d65419096bb-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:08.346702 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.346653 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" event={"ID":"415a9325-f24b-4704-9204-1ff2cb43d51d","Type":"ContainerDied","Data":"79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa"} Jul 28 19:44:08.346702 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.346698 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr" Jul 28 19:44:08.347155 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.346716 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b920d912c51b0a7022ba3f3ac3952cbaae37d9ca5edf278bf6a9ebf30031aa" Jul 28 19:44:08.348625 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.348606 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" Jul 28 19:44:08.348750 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.348649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw" event={"ID":"53aad1db-8cad-44da-be15-9d65419096bb","Type":"ContainerDied","Data":"dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28"} Jul 28 19:44:08.348750 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.348716 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad6e63b52d0420e17ce62fb95406de23a3de8a5ae416c2ad9cb9a73e6822d28" Jul 28 19:44:08.350520 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.350499 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" Jul 28 19:44:08.350520 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.350516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz" event={"ID":"f36cc7cf-d61a-4270-9199-baeb90d78dbe","Type":"ContainerDied","Data":"7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb"} Jul 28 19:44:08.350712 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.350546 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7099b09d9c15cd5779b574bb800560c2a58f6ebd1849616b359bca950183c7fb" Jul 28 19:44:08.483785 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.483765 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:08.621686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.621595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle\") pod \"271d3d76-5efa-4337-9fe9-def48a6ee259\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " Jul 28 19:44:08.621686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.621646 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util\") pod \"271d3d76-5efa-4337-9fe9-def48a6ee259\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " Jul 28 19:44:08.621891 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.621736 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhl2g\" (UniqueName: \"kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g\") pod \"271d3d76-5efa-4337-9fe9-def48a6ee259\" (UID: \"271d3d76-5efa-4337-9fe9-def48a6ee259\") " Jul 28 19:44:08.622131 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.622097 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle" (OuterVolumeSpecName: "bundle") pod "271d3d76-5efa-4337-9fe9-def48a6ee259" (UID: "271d3d76-5efa-4337-9fe9-def48a6ee259"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:08.623810 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.623787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g" (OuterVolumeSpecName: "kube-api-access-qhl2g") pod "271d3d76-5efa-4337-9fe9-def48a6ee259" (UID: "271d3d76-5efa-4337-9fe9-def48a6ee259"). InnerVolumeSpecName "kube-api-access-qhl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:08.626938 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.626904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util" (OuterVolumeSpecName: "util") pod "271d3d76-5efa-4337-9fe9-def48a6ee259" (UID: "271d3d76-5efa-4337-9fe9-def48a6ee259"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:08.723021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.722987 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-bundle\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:08.723021 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.723025 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/271d3d76-5efa-4337-9fe9-def48a6ee259-util\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:08.723178 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:08.723039 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhl2g\" (UniqueName: \"kubernetes.io/projected/271d3d76-5efa-4337-9fe9-def48a6ee259-kube-api-access-qhl2g\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:09.358816 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:09.358780 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" event={"ID":"271d3d76-5efa-4337-9fe9-def48a6ee259","Type":"ContainerDied","Data":"77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8"} Jul 28 19:44:09.359170 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:09.358835 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c0514be3c8a196345879337267e9f3902333cbbdcc2d7f8f1b67200aab66a8" Jul 28 19:44:09.359170 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:09.358839 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj" Jul 28 19:44:18.532930 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.532882 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m"] Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533214 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533228 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533239 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533247 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533260 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533269 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533281 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533288 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533316 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533322 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533330 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533338 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533354 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533362 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="util" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533370 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533378 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533387 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533394 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533403 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533410 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="extract" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533417 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="pull" Jul 28 19:44:18.533417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533425 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="pull" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533437 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="extract" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533445 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="extract" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533534 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36cc7cf-d61a-4270-9199-baeb90d78dbe" containerName="extract" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533547 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53aad1db-8cad-44da-be15-9d65419096bb" containerName="extract" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533557 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="271d3d76-5efa-4337-9fe9-def48a6ee259" containerName="extract" Jul 28 19:44:18.534079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.533566 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="415a9325-f24b-4704-9204-1ff2cb43d51d" containerName="extract" Jul 28 19:44:18.540071 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.540051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:18.543337 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.543314 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 28 19:44:18.543448 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.543388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kcqhm\"" Jul 28 19:44:18.554866 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.554842 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m"] Jul 28 19:44:18.696334 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.696291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ch4\" (UniqueName: \"kubernetes.io/projected/ead17357-921b-4f93-9593-d7f44df3e381-kube-api-access-l5ch4\") pod \"dns-operator-controller-manager-864dd78c4c-ltg8m\" (UID: \"ead17357-921b-4f93-9593-d7f44df3e381\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:18.797657 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.797550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ch4\" (UniqueName: \"kubernetes.io/projected/ead17357-921b-4f93-9593-d7f44df3e381-kube-api-access-l5ch4\") pod \"dns-operator-controller-manager-864dd78c4c-ltg8m\" (UID: \"ead17357-921b-4f93-9593-d7f44df3e381\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:18.808149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.808121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ch4\" (UniqueName: \"kubernetes.io/projected/ead17357-921b-4f93-9593-d7f44df3e381-kube-api-access-l5ch4\") pod \"dns-operator-controller-manager-864dd78c4c-ltg8m\" (UID: \"ead17357-921b-4f93-9593-d7f44df3e381\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:18.849761 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.849729 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:18.984633 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:18.984608 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m"] Jul 28 19:44:19.410338 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:19.410306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" event={"ID":"ead17357-921b-4f93-9593-d7f44df3e381","Type":"ContainerStarted","Data":"d132997257b4303e6ad601b8f5d51bf19974706a832feb0730423a7c7696840b"} Jul 28 19:44:21.421396 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:21.421360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" event={"ID":"ead17357-921b-4f93-9593-d7f44df3e381","Type":"ContainerStarted","Data":"3c5b61bebde138ca787cb2e1be748448457d7ed60057eb02075c7a5ab2f8947e"} Jul 28 19:44:21.421872 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:21.421453 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:21.438969 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:21.438920 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" podStartSLOduration=1.648288813 podStartE2EDuration="3.438907075s" totalImagesPullingTime="1.790618262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:21.437566539 +0000 UTC m=+480.873520491" watchObservedRunningTime="2026-07-28 19:44:21.438907075 +0000 UTC m=+480.874861027" Jul 28 19:44:22.148656 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.148622 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz"] Jul 28 19:44:22.153893 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.153858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:22.156337 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.156319 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dbs6w\"" Jul 28 19:44:22.165413 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.165390 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz"] Jul 28 19:44:22.327429 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.327393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzp2\" (UniqueName: \"kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2\") pod \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" (UID: \"21e8d8bb-0461-4849-ba51-f11d8e4f2195\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:22.427979 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.427951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzp2\" (UniqueName: \"kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2\") pod \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" (UID: \"21e8d8bb-0461-4849-ba51-f11d8e4f2195\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:22.441651 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.441626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzp2\" (UniqueName: \"kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2\") pod \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" (UID: \"21e8d8bb-0461-4849-ba51-f11d8e4f2195\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:22.465624 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.465599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:22.607606 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:22.607578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz"] Jul 28 19:44:22.609231 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:22.609205 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e8d8bb_0461_4849_ba51_f11d8e4f2195.slice/crio-6286165a2f894e851c013846dd2420d1f7086b1e68ba8d477b7cbc4218e0cc24 WatchSource:0}: Error finding container 6286165a2f894e851c013846dd2420d1f7086b1e68ba8d477b7cbc4218e0cc24: Status 404 returned error can't find the container with id 6286165a2f894e851c013846dd2420d1f7086b1e68ba8d477b7cbc4218e0cc24 Jul 28 19:44:23.434114 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:23.434038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" event={"ID":"21e8d8bb-0461-4849-ba51-f11d8e4f2195","Type":"ContainerStarted","Data":"6286165a2f894e851c013846dd2420d1f7086b1e68ba8d477b7cbc4218e0cc24"} Jul 28 19:44:24.440581 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:24.440541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" event={"ID":"21e8d8bb-0461-4849-ba51-f11d8e4f2195","Type":"ContainerStarted","Data":"68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb"} Jul 28 19:44:24.440961 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:24.440613 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:24.456953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:24.456902 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" podStartSLOduration=1.029998008 podStartE2EDuration="2.45688692s" totalImagesPullingTime="1.426888912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:24.456185358 +0000 UTC m=+483.892139312" watchObservedRunningTime="2026-07-28 19:44:24.45688692 +0000 UTC m=+483.892840920" Jul 28 19:44:26.903924 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:26.903886 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82"] Jul 28 19:44:26.907479 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:26.907454 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:26.910696 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:26.910675 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8qgj2\"" Jul 28 19:44:26.919233 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:26.919212 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82"] Jul 28 19:44:27.070842 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.070805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.071006 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.070885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblrp\" (UniqueName: \"kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.172182 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.172103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblrp\" (UniqueName: \"kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.172182 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.172151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.172476 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.172459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.184359 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.184340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblrp\" (UniqueName: \"kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.219374 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.219349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:27.365100 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.365071 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82"] Jul 28 19:44:27.366758 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:27.366728 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4de021_9aed_4561_9c91_8e60de0c0904.slice/crio-156d1309c95f7b05370e3f1720dda5fefe84cec89a59373f8012b45df51c8a1f WatchSource:0}: Error finding container 156d1309c95f7b05370e3f1720dda5fefe84cec89a59373f8012b45df51c8a1f: Status 404 returned error can't find the container with id 156d1309c95f7b05370e3f1720dda5fefe84cec89a59373f8012b45df51c8a1f Jul 28 19:44:27.458149 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:27.458078 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" event={"ID":"bd4de021-9aed-4561-9c91-8e60de0c0904","Type":"ContainerStarted","Data":"156d1309c95f7b05370e3f1720dda5fefe84cec89a59373f8012b45df51c8a1f"} Jul 28 19:44:30.240477 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.240447 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prsdm"] Jul 28 19:44:30.244209 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.244185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:30.248287 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.248259 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-d4jnc\"" Jul 28 19:44:30.255285 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.255254 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prsdm"] Jul 28 19:44:30.402923 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.402886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjw5\" (UniqueName: \"kubernetes.io/projected/d4e17283-4328-4fbf-a98b-ac3fe8779270-kube-api-access-9gjw5\") pod \"authorino-operator-75cc8d5cb7-prsdm\" (UID: \"d4e17283-4328-4fbf-a98b-ac3fe8779270\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:30.503759 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.503654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjw5\" (UniqueName: \"kubernetes.io/projected/d4e17283-4328-4fbf-a98b-ac3fe8779270-kube-api-access-9gjw5\") pod \"authorino-operator-75cc8d5cb7-prsdm\" (UID: \"d4e17283-4328-4fbf-a98b-ac3fe8779270\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:30.522140 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.522102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjw5\" (UniqueName: \"kubernetes.io/projected/d4e17283-4328-4fbf-a98b-ac3fe8779270-kube-api-access-9gjw5\") pod \"authorino-operator-75cc8d5cb7-prsdm\" (UID: \"d4e17283-4328-4fbf-a98b-ac3fe8779270\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:30.558451 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.558415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:30.708752 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:30.707559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prsdm"] Jul 28 19:44:31.482613 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:31.482572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" event={"ID":"d4e17283-4328-4fbf-a98b-ac3fe8779270","Type":"ContainerStarted","Data":"ef3de3779a29b7cf37eb81a85707d7e41b6aaf50f20a2c19de7a606db0b64b5a"} Jul 28 19:44:32.429511 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:32.429470 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltg8m" Jul 28 19:44:32.491146 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:32.491108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" event={"ID":"bd4de021-9aed-4561-9c91-8e60de0c0904","Type":"ContainerStarted","Data":"80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b"} Jul 28 19:44:32.491537 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:32.491240 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:32.511887 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:32.511825 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" podStartSLOduration=1.871340275 podStartE2EDuration="6.511804932s" totalImagesPullingTime="4.640464657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:32.510395751 +0000 UTC m=+491.946349697" watchObservedRunningTime="2026-07-28 19:44:32.511804932 +0000 UTC m=+491.947758886" Jul 28 19:44:34.502449 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.502415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" event={"ID":"d4e17283-4328-4fbf-a98b-ac3fe8779270","Type":"ContainerStarted","Data":"6fdd5ad633f79e3884b87de9e1edf7735d3349c1415c57c0ab284ecfb7bbd148"} Jul 28 19:44:34.502869 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.502487 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:34.519772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.519725 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" podStartSLOduration=1.425308863 podStartE2EDuration="4.519710826s" totalImagesPullingTime="3.094401963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:34.519080754 +0000 UTC m=+493.955034706" watchObservedRunningTime="2026-07-28 19:44:34.519710826 +0000 UTC m=+493.955664778" Jul 28 19:44:34.887050 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.886963 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj"] Jul 28 19:44:34.890517 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.890493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:34.893064 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.893043 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 28 19:44:34.893175 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.893047 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q24p6\"" Jul 28 19:44:34.893175 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.893114 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 28 19:44:34.899023 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:34.899001 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj"] Jul 28 19:44:35.046061 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.046031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pp6\" (UniqueName: \"kubernetes.io/projected/d096c60c-e8e3-427f-aeb9-a15d1036f0da-kube-api-access-f7pp6\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.046256 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.046069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d096c60c-e8e3-427f-aeb9-a15d1036f0da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.046256 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.046108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d096c60c-e8e3-427f-aeb9-a15d1036f0da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.147651 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.147573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pp6\" (UniqueName: \"kubernetes.io/projected/d096c60c-e8e3-427f-aeb9-a15d1036f0da-kube-api-access-f7pp6\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.147651 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.147614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d096c60c-e8e3-427f-aeb9-a15d1036f0da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.147844 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.147740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d096c60c-e8e3-427f-aeb9-a15d1036f0da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.148338 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.148315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d096c60c-e8e3-427f-aeb9-a15d1036f0da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.150008 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.149977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d096c60c-e8e3-427f-aeb9-a15d1036f0da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.155564 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.155541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pp6\" (UniqueName: \"kubernetes.io/projected/d096c60c-e8e3-427f-aeb9-a15d1036f0da-kube-api-access-f7pp6\") pod \"kuadrant-console-plugin-66b5774985-lxlmj\" (UID: \"d096c60c-e8e3-427f-aeb9-a15d1036f0da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.200487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.200443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" Jul 28 19:44:35.333310 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.333287 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj"] Jul 28 19:44:35.335003 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:35.334973 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd096c60c_e8e3_427f_aeb9_a15d1036f0da.slice/crio-9067cbeac67a754a84c0f5446c5f88cca085ecb93e3a12484084562719b673ed WatchSource:0}: Error finding container 9067cbeac67a754a84c0f5446c5f88cca085ecb93e3a12484084562719b673ed: Status 404 returned error can't find the container with id 9067cbeac67a754a84c0f5446c5f88cca085ecb93e3a12484084562719b673ed Jul 28 19:44:35.447897 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.447871 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:35.510540 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:35.510506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" event={"ID":"d096c60c-e8e3-427f-aeb9-a15d1036f0da","Type":"ContainerStarted","Data":"9067cbeac67a754a84c0f5446c5f88cca085ecb93e3a12484084562719b673ed"} Jul 28 19:44:43.498815 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:43.498779 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:45.426148 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.426114 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82"] Jul 28 19:44:45.426602 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.426357 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" containerName="manager" containerID="cri-o://80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b" gracePeriod=2 Jul 28 19:44:45.441937 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.441902 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:44:45.448276 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.448250 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82"] Jul 28 19:44:45.448449 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.448328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.450910 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.450874 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bd4de021-9aed-4561-9c91-8e60de0c0904\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jsc82\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:45.460866 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.460837 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz"] Jul 28 19:44:45.461135 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.461100 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" containerName="manager" containerID="cri-o://68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb" gracePeriod=2 Jul 28 19:44:45.467105 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.467063 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:44:45.467609 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.467586 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" containerName="manager" Jul 28 19:44:45.467609 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.467612 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" containerName="manager" Jul 28 19:44:45.467776 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.467745 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" containerName="manager" Jul 28 19:44:45.472467 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.472390 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:44:45.472574 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.472493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.473583 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.473556 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz"] Jul 28 19:44:45.483214 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.483179 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:45.487078 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.487044 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:45.489394 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.489364 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:45.490022 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.489996 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:44:45.514013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.513982 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prsdm" Jul 28 19:44:45.516798 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.516755 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:45.542986 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.542956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.544090 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.543031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztd48\" (UniqueName: \"kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.544090 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.543072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkzl\" (UniqueName: \"kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.544090 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.543195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.545758 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:45.545726 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:45.644123 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkzl\" (UniqueName: \"kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.644335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.644335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.644335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd48\" (UniqueName: \"kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.644619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.644713 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.644603 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.653857 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.653829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkzl\" (UniqueName: \"kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lr756\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.653992 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.653827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd48\" (UniqueName: \"kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rzv9d\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:45.848870 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.848794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:45.856470 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:45.856439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:56.977796 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:56.977747 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:56.980579 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:56.980543 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:56.983982 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:56.983843 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:56.985990 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:56.985958 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:56.987784 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:56.987751 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:57.018240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.018215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:44:57.019963 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:57.019931 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6064de25_c7b8_46eb_a06b_426fbddc63e4.slice/crio-b88c66e14be6eddabb527647230a7f8b5ab6e80c78b2b7236591749375ba05e8 WatchSource:0}: Error finding container b88c66e14be6eddabb527647230a7f8b5ab6e80c78b2b7236591749375ba05e8: Status 404 returned error can't find the container with id b88c66e14be6eddabb527647230a7f8b5ab6e80c78b2b7236591749375ba05e8 Jul 28 19:44:57.051087 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.051063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qblrp\" (UniqueName: \"kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp\") pod \"bd4de021-9aed-4561-9c91-8e60de0c0904\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " Jul 28 19:44:57.051209 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.051107 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume\") pod \"bd4de021-9aed-4561-9c91-8e60de0c0904\" (UID: \"bd4de021-9aed-4561-9c91-8e60de0c0904\") " Jul 28 19:44:57.051209 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.051184 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgzp2\" (UniqueName: \"kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2\") pod \"21e8d8bb-0461-4849-ba51-f11d8e4f2195\" (UID: \"21e8d8bb-0461-4849-ba51-f11d8e4f2195\") " Jul 28 19:44:57.051757 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.051726 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bd4de021-9aed-4561-9c91-8e60de0c0904" (UID: "bd4de021-9aed-4561-9c91-8e60de0c0904"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:44:57.053043 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.053012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp" (OuterVolumeSpecName: "kube-api-access-qblrp") pod "bd4de021-9aed-4561-9c91-8e60de0c0904" (UID: "bd4de021-9aed-4561-9c91-8e60de0c0904"). InnerVolumeSpecName "kube-api-access-qblrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:57.053157 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.053097 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2" (OuterVolumeSpecName: "kube-api-access-fgzp2") pod "21e8d8bb-0461-4849-ba51-f11d8e4f2195" (UID: "21e8d8bb-0461-4849-ba51-f11d8e4f2195"). InnerVolumeSpecName "kube-api-access-fgzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:44:57.152017 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.151992 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qblrp\" (UniqueName: \"kubernetes.io/projected/bd4de021-9aed-4561-9c91-8e60de0c0904-kube-api-access-qblrp\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:57.152017 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.152019 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4de021-9aed-4561-9c91-8e60de0c0904-extensions-socket-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:57.152197 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.152030 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzp2\" (UniqueName: \"kubernetes.io/projected/21e8d8bb-0461-4849-ba51-f11d8e4f2195-kube-api-access-fgzp2\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:44:57.165200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.165169 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" path="/var/lib/kubelet/pods/21e8d8bb-0461-4849-ba51-f11d8e4f2195/volumes" Jul 28 19:44:57.165526 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.165514 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" path="/var/lib/kubelet/pods/bd4de021-9aed-4561-9c91-8e60de0c0904/volumes" Jul 28 19:44:57.244769 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.244745 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:44:57.247333 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:44:57.247308 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e818d4_32b4_427e_be58_fceec3815eac.slice/crio-fd2d709d67ad196b911b161e8ebdabf27a23cf4692343a9f2dff73fad8eb75ed WatchSource:0}: Error finding container fd2d709d67ad196b911b161e8ebdabf27a23cf4692343a9f2dff73fad8eb75ed: Status 404 returned error can't find the container with id fd2d709d67ad196b911b161e8ebdabf27a23cf4692343a9f2dff73fad8eb75ed Jul 28 19:44:57.656355 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.656317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" event={"ID":"d096c60c-e8e3-427f-aeb9-a15d1036f0da","Type":"ContainerStarted","Data":"8d2ae6978f0098e1e638b08d4bb8f10ae5892a429601945073c65269cab83790"} Jul 28 19:44:57.658057 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.658032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" event={"ID":"62e818d4-32b4-427e-be58-fceec3815eac","Type":"ContainerStarted","Data":"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f"} Jul 28 19:44:57.658188 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.658072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" event={"ID":"62e818d4-32b4-427e-be58-fceec3815eac","Type":"ContainerStarted","Data":"fd2d709d67ad196b911b161e8ebdabf27a23cf4692343a9f2dff73fad8eb75ed"} Jul 28 19:44:57.658188 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.658113 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:44:57.659301 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.659281 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd4de021-9aed-4561-9c91-8e60de0c0904" containerID="80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b" exitCode=0 Jul 28 19:44:57.659424 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.659315 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:57.659424 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.659374 2562 scope.go:117] "RemoveContainer" containerID="80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b" Jul 28 19:44:57.661236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.661211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" event={"ID":"6064de25-c7b8-46eb-a06b-426fbddc63e4","Type":"ContainerStarted","Data":"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31"} Jul 28 19:44:57.661402 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.661386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" event={"ID":"6064de25-c7b8-46eb-a06b-426fbddc63e4","Type":"ContainerStarted","Data":"b88c66e14be6eddabb527647230a7f8b5ab6e80c78b2b7236591749375ba05e8"} Jul 28 19:44:57.661501 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.661441 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:44:57.662600 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.662577 2562 generic.go:356] "Generic (PLEG): container finished" podID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" containerID="68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb" exitCode=0 Jul 28 19:44:57.662724 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.662617 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:57.669139 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.669123 2562 scope.go:117] "RemoveContainer" containerID="80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b" Jul 28 19:44:57.669379 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:57.669364 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b\": container with ID starting with 80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b not found: ID does not exist" containerID="80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b" Jul 28 19:44:57.669417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.669387 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b"} err="failed to get container status \"80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b\": rpc error: code = NotFound desc = could not find container \"80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b\": container with ID starting with 80da1abf6cf511656b173b4648d4dbf72344adbceea7821c3e0c26058a31c32b not found: ID does not exist" Jul 28 19:44:57.669417 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.669404 2562 scope.go:117] "RemoveContainer" containerID="68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb" Jul 28 19:44:57.674834 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:57.674812 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mqgvz\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mqgvz" Jul 28 19:44:57.675612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.675570 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lxlmj" podStartSLOduration=2.05219876 podStartE2EDuration="23.675553629s" totalImagesPullingTime="21.623354869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:57.672630756 +0000 UTC m=+517.108584710" watchObservedRunningTime="2026-07-28 19:44:57.675553629 +0000 UTC m=+517.111507582" Jul 28 19:44:57.678703 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.678686 2562 scope.go:117] "RemoveContainer" containerID="68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb" Jul 28 19:44:57.678947 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:57.678932 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb\": container with ID starting with 68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb not found: ID does not exist" containerID="68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb" Jul 28 19:44:57.678989 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.678953 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb"} err="failed to get container status \"68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb\": rpc error: code = NotFound desc = could not find container \"68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb\": container with ID starting with 68267b845c4d60faad5a8ddf2f77474e4117854b5e2fe401ca2cb31c73423dcb not found: ID does not exist" Jul 28 19:44:57.694725 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.694680 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" podStartSLOduration=12.694653237 podStartE2EDuration="12.694653237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:57.692968517 +0000 UTC m=+517.128922470" watchObservedRunningTime="2026-07-28 19:44:57.694653237 +0000 UTC m=+517.130607189" Jul 28 19:44:57.695003 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:44:57.694973 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jsc82\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="bd4de021-9aed-4561-9c91-8e60de0c0904" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jsc82" Jul 28 19:44:57.715720 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:44:57.715655 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" podStartSLOduration=12.715643222 podStartE2EDuration="12.715643222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:44:57.714423428 +0000 UTC m=+517.150377392" watchObservedRunningTime="2026-07-28 19:44:57.715643222 +0000 UTC m=+517.151597173" Jul 28 19:45:08.672509 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:08.672474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:45:08.672915 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:08.672553 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:45:08.752998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:08.752966 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:45:08.753166 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:08.753143 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" podUID="62e818d4-32b4-427e-be58-fceec3815eac" containerName="manager" containerID="cri-o://315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f" gracePeriod=10 Jul 28 19:45:09.007553 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.007531 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:45:09.063286 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063260 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:45:09.063604 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063592 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" containerName="manager" Jul 28 19:45:09.063643 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063606 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" containerName="manager" Jul 28 19:45:09.063643 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063632 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e818d4-32b4-427e-be58-fceec3815eac" containerName="manager" Jul 28 19:45:09.063643 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063637 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e818d4-32b4-427e-be58-fceec3815eac" containerName="manager" Jul 28 19:45:09.063750 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063704 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e818d4-32b4-427e-be58-fceec3815eac" containerName="manager" Jul 28 19:45:09.063750 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.063713 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e8d8bb-0461-4849-ba51-f11d8e4f2195" containerName="manager" Jul 28 19:45:09.065543 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.065527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.082026 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.081998 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:45:09.153389 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.153353 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume\") pod \"62e818d4-32b4-427e-be58-fceec3815eac\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " Jul 28 19:45:09.153562 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.153403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztd48\" (UniqueName: \"kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48\") pod \"62e818d4-32b4-427e-be58-fceec3815eac\" (UID: \"62e818d4-32b4-427e-be58-fceec3815eac\") " Jul 28 19:45:09.153603 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.153569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqv7k\" (UniqueName: \"kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.153636 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.153611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.153829 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.153795 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "62e818d4-32b4-427e-be58-fceec3815eac" (UID: "62e818d4-32b4-427e-be58-fceec3815eac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:45:09.155486 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.155466 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48" (OuterVolumeSpecName: "kube-api-access-ztd48") pod "62e818d4-32b4-427e-be58-fceec3815eac" (UID: "62e818d4-32b4-427e-be58-fceec3815eac"). InnerVolumeSpecName "kube-api-access-ztd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:45:09.254732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.254599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqv7k\" (UniqueName: \"kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.254732 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.254696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.254944 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.254774 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62e818d4-32b4-427e-be58-fceec3815eac-extensions-socket-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:09.254944 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.254787 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztd48\" (UniqueName: \"kubernetes.io/projected/62e818d4-32b4-427e-be58-fceec3815eac-kube-api-access-ztd48\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:09.255114 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.255094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.265085 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.265058 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqv7k\" (UniqueName: \"kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k\") pod \"kuadrant-operator-controller-manager-86878c769f-lhd7b\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.374619 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.374584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:09.516200 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.516170 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:45:09.518392 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:45:09.518369 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537e011a_adf5_43a9_b602_f8024fd02244.slice/crio-308f8347fa65b354b7396becb2f94b2a7ab1ffb0a2d1a4754d1e93ec1b220c29 WatchSource:0}: Error finding container 308f8347fa65b354b7396becb2f94b2a7ab1ffb0a2d1a4754d1e93ec1b220c29: Status 404 returned error can't find the container with id 308f8347fa65b354b7396becb2f94b2a7ab1ffb0a2d1a4754d1e93ec1b220c29 Jul 28 19:45:09.733814 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.733777 2562 generic.go:356] "Generic (PLEG): container finished" podID="62e818d4-32b4-427e-be58-fceec3815eac" containerID="315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f" exitCode=0 Jul 28 19:45:09.734247 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.733849 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" Jul 28 19:45:09.734247 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.733860 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" event={"ID":"62e818d4-32b4-427e-be58-fceec3815eac","Type":"ContainerDied","Data":"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f"} Jul 28 19:45:09.734247 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.733955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d" event={"ID":"62e818d4-32b4-427e-be58-fceec3815eac","Type":"ContainerDied","Data":"fd2d709d67ad196b911b161e8ebdabf27a23cf4692343a9f2dff73fad8eb75ed"} Jul 28 19:45:09.734247 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.733991 2562 scope.go:117] "RemoveContainer" containerID="315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f" Jul 28 19:45:09.735818 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.735793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" event={"ID":"537e011a-adf5-43a9-b602-f8024fd02244","Type":"ContainerStarted","Data":"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24"} Jul 28 19:45:09.735950 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.735838 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" event={"ID":"537e011a-adf5-43a9-b602-f8024fd02244","Type":"ContainerStarted","Data":"308f8347fa65b354b7396becb2f94b2a7ab1ffb0a2d1a4754d1e93ec1b220c29"} Jul 28 19:45:09.743040 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.743024 2562 scope.go:117] "RemoveContainer" containerID="315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f" Jul 28 19:45:09.743304 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:45:09.743281 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f\": container with ID starting with 315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f not found: ID does not exist" containerID="315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f" Jul 28 19:45:09.743399 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.743308 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f"} err="failed to get container status \"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f\": rpc error: code = NotFound desc = could not find container \"315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f\": container with ID starting with 315abb28af1bae1e30721f1b71f9993b71e93b6fc01bcf06f7972bb20156ee8f not found: ID does not exist" Jul 28 19:45:09.759427 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.759374 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" podStartSLOduration=0.75936047 podStartE2EDuration="759.36047ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:09.758685875 +0000 UTC m=+529.194639828" watchObservedRunningTime="2026-07-28 19:45:09.75936047 +0000 UTC m=+529.195314418" Jul 28 19:45:09.775803 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.775753 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:45:09.780471 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:09.780452 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rzv9d"] Jul 28 19:45:10.742218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:10.742191 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:11.163005 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:11.162973 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e818d4-32b4-427e-be58-fceec3815eac" path="/var/lib/kubelet/pods/62e818d4-32b4-427e-be58-fceec3815eac/volumes" Jul 28 19:45:21.749624 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:21.749596 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:45:21.806191 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:21.806160 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:45:21.806370 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:21.806345 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" podUID="6064de25-c7b8-46eb-a06b-426fbddc63e4" containerName="manager" containerID="cri-o://ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31" gracePeriod=10 Jul 28 19:45:22.069734 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.069705 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:45:22.164584 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.164548 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwkzl\" (UniqueName: \"kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl\") pod \"6064de25-c7b8-46eb-a06b-426fbddc63e4\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " Jul 28 19:45:22.164772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.164628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume\") pod \"6064de25-c7b8-46eb-a06b-426fbddc63e4\" (UID: \"6064de25-c7b8-46eb-a06b-426fbddc63e4\") " Jul 28 19:45:22.164964 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.164938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6064de25-c7b8-46eb-a06b-426fbddc63e4" (UID: "6064de25-c7b8-46eb-a06b-426fbddc63e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:45:22.166655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.166632 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl" (OuterVolumeSpecName: "kube-api-access-jwkzl") pod "6064de25-c7b8-46eb-a06b-426fbddc63e4" (UID: "6064de25-c7b8-46eb-a06b-426fbddc63e4"). InnerVolumeSpecName "kube-api-access-jwkzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:45:22.266235 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.266197 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6064de25-c7b8-46eb-a06b-426fbddc63e4-extensions-socket-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:22.266235 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.266230 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkzl\" (UniqueName: \"kubernetes.io/projected/6064de25-c7b8-46eb-a06b-426fbddc63e4-kube-api-access-jwkzl\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:22.811443 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.811391 2562 generic.go:356] "Generic (PLEG): container finished" podID="6064de25-c7b8-46eb-a06b-426fbddc63e4" containerID="ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31" exitCode=0 Jul 28 19:45:22.811443 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.811441 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" Jul 28 19:45:22.811963 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.811458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" event={"ID":"6064de25-c7b8-46eb-a06b-426fbddc63e4","Type":"ContainerDied","Data":"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31"} Jul 28 19:45:22.811963 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.811508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756" event={"ID":"6064de25-c7b8-46eb-a06b-426fbddc63e4","Type":"ContainerDied","Data":"b88c66e14be6eddabb527647230a7f8b5ab6e80c78b2b7236591749375ba05e8"} Jul 28 19:45:22.811963 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.811532 2562 scope.go:117] "RemoveContainer" containerID="ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31" Jul 28 19:45:22.821733 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.821716 2562 scope.go:117] "RemoveContainer" containerID="ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31" Jul 28 19:45:22.821947 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:45:22.821930 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31\": container with ID starting with ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31 not found: ID does not exist" containerID="ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31" Jul 28 19:45:22.821997 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.821956 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31"} err="failed to get container status \"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31\": rpc error: code = NotFound desc = could not find container \"ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31\": container with ID starting with ceb8dba4bf8ab85cac83e893f3376f19385cf7e22dcc8382e30c4504bca86c31 not found: ID does not exist" Jul 28 19:45:22.835971 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.835950 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:45:22.842070 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:22.842048 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lr756"] Jul 28 19:45:23.163100 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:23.163069 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6064de25-c7b8-46eb-a06b-426fbddc63e4" path="/var/lib/kubelet/pods/6064de25-c7b8-46eb-a06b-426fbddc63e4/volumes" Jul 28 19:45:25.066290 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.066252 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd"] Jul 28 19:45:25.066803 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.066787 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6064de25-c7b8-46eb-a06b-426fbddc63e4" containerName="manager" Jul 28 19:45:25.066803 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.066804 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6064de25-c7b8-46eb-a06b-426fbddc63e4" containerName="manager" Jul 28 19:45:25.066905 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.066877 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6064de25-c7b8-46eb-a06b-426fbddc63e4" containerName="manager" Jul 28 19:45:25.144954 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.144916 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd"] Jul 28 19:45:25.145100 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.145012 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.147724 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.147701 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 19:45:25.147879 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.147827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sphr9\"" Jul 28 19:45:25.290369 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8wr\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-kube-api-access-5n8wr\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290392 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290519 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.290655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.290645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391366 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391366 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391366 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8wr\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-kube-api-access-5n8wr\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391620 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391967 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391742 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.391967 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.392074 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.391979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.392074 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.392026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.392147 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.392078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.393862 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.393840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.394115 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.394096 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.399655 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.399632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.399998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.399978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8wr\" (UniqueName: \"kubernetes.io/projected/2282a22c-3b4f-4477-a047-13cd7ebb7d6e-kube-api-access-5n8wr\") pod \"maas-default-gateway-openshift-default-5ccb968c66-f9dwd\" (UID: \"2282a22c-3b4f-4477-a047-13cd7ebb7d6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.456141 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.456117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:25.617473 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.617450 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd"] Jul 28 19:45:25.620064 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:45:25.620036 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2282a22c_3b4f_4477_a047_13cd7ebb7d6e.slice/crio-0eb4e22d85e9475cdf1e5aaa0bce109b7ea653367e9f03b77cf050549cf18dca WatchSource:0}: Error finding container 0eb4e22d85e9475cdf1e5aaa0bce109b7ea653367e9f03b77cf050549cf18dca: Status 404 returned error can't find the container with id 0eb4e22d85e9475cdf1e5aaa0bce109b7ea653367e9f03b77cf050549cf18dca Jul 28 19:45:25.829524 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:25.829487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" event={"ID":"2282a22c-3b4f-4477-a047-13cd7ebb7d6e","Type":"ContainerStarted","Data":"0eb4e22d85e9475cdf1e5aaa0bce109b7ea653367e9f03b77cf050549cf18dca"} Jul 28 19:45:28.853680 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:28.853619 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:45:28.853966 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:28.853729 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:45:28.853966 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:28.853763 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:45:29.852112 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:29.852077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" event={"ID":"2282a22c-3b4f-4477-a047-13cd7ebb7d6e","Type":"ContainerStarted","Data":"0f285b66a2617f676c774eeb47da0631904af544ba24c67f232b0141fc17da2f"} Jul 28 19:45:29.875128 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:29.875072 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" podStartSLOduration=1.644082376 podStartE2EDuration="4.87505449s" totalImagesPullingTime="3.230972114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:29.87313928 +0000 UTC m=+549.309093232" watchObservedRunningTime="2026-07-28 19:45:29.87505449 +0000 UTC m=+549.311008441" Jul 28 19:45:30.456830 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:30.456794 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:30.462345 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:30.462321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:30.858587 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:30.858496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:30.859413 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:30.859393 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:45:39.302943 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.302907 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:39.307410 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.307389 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.309941 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.309921 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 19:45:39.314219 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.314196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:39.407497 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.407460 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:39.407893 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.407848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktp5\" (UniqueName: \"kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.408031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.408013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.508501 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.508466 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.508684 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.508548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktp5\" (UniqueName: \"kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.509143 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.509123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.518612 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.518582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktp5\" (UniqueName: \"kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5\") pod \"limitador-limitador-64c94fd64f-zqxjm\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.620727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.620642 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:39.780375 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.780328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:39.783560 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:45:39.783534 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d91c741_89f9_4d75_97e2_e957f7029faf.slice/crio-023d74fda4c2ef5096d3934628480814081784d0f1c75e2b70fcc99a86377279 WatchSource:0}: Error finding container 023d74fda4c2ef5096d3934628480814081784d0f1c75e2b70fcc99a86377279: Status 404 returned error can't find the container with id 023d74fda4c2ef5096d3934628480814081784d0f1c75e2b70fcc99a86377279 Jul 28 19:45:39.916092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:39.916051 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" event={"ID":"7d91c741-89f9-4d75-97e2-e957f7029faf","Type":"ContainerStarted","Data":"023d74fda4c2ef5096d3934628480814081784d0f1c75e2b70fcc99a86377279"} Jul 28 19:45:40.231789 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.231712 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:45:40.236494 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.236470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:45:40.239039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.239012 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grx9k\"" Jul 28 19:45:40.241928 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.241903 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:45:40.316522 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.316490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxsw\" (UniqueName: \"kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw\") pod \"authorino-75d556bd77-d8kmb\" (UID: \"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114\") " pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:45:40.418093 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.418051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxsw\" (UniqueName: \"kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw\") pod \"authorino-75d556bd77-d8kmb\" (UID: \"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114\") " pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:45:40.427622 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.427584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxsw\" (UniqueName: \"kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw\") pod \"authorino-75d556bd77-d8kmb\" (UID: \"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114\") " pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:45:40.547043 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.546955 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:45:40.750774 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.750727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:45:40.751876 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:45:40.751845 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa7ee9c3_f6f8_464f_aa59_8fcf9caf4114.slice/crio-5790e7ebb44c76fa3d64ab8991b04bfaff23ea47068356d4ce80793bc68ab964 WatchSource:0}: Error finding container 5790e7ebb44c76fa3d64ab8991b04bfaff23ea47068356d4ce80793bc68ab964: Status 404 returned error can't find the container with id 5790e7ebb44c76fa3d64ab8991b04bfaff23ea47068356d4ce80793bc68ab964 Jul 28 19:45:40.924012 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:40.923979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d8kmb" event={"ID":"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114","Type":"ContainerStarted","Data":"5790e7ebb44c76fa3d64ab8991b04bfaff23ea47068356d4ce80793bc68ab964"} Jul 28 19:45:42.938707 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:42.938592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" event={"ID":"7d91c741-89f9-4d75-97e2-e957f7029faf","Type":"ContainerStarted","Data":"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758"} Jul 28 19:45:42.939141 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:42.938796 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:42.957422 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:42.957374 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" podStartSLOduration=1.160220488 podStartE2EDuration="3.957361192s" totalImagesPullingTime="2.797140704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:42.954799419 +0000 UTC m=+562.390753372" watchObservedRunningTime="2026-07-28 19:45:42.957361192 +0000 UTC m=+562.393315143" Jul 28 19:45:44.953602 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:44.953568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d8kmb" event={"ID":"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114","Type":"ContainerStarted","Data":"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e"} Jul 28 19:45:44.968524 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:44.968474 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-d8kmb" podStartSLOduration=0.865739854 podStartE2EDuration="4.968456939s" totalImagesPullingTime="4.102717085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:44.967871316 +0000 UTC m=+564.403825268" watchObservedRunningTime="2026-07-28 19:45:44.968456939 +0000 UTC m=+564.404410920" Jul 28 19:45:53.946372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:53.946335 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:56.248244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.248207 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:56.248736 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.248383 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" podUID="7d91c741-89f9-4d75-97e2-e957f7029faf" containerName="limitador" containerID="cri-o://347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758" gracePeriod=30 Jul 28 19:45:56.809622 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.809564 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:56.858728 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.858699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sktp5\" (UniqueName: \"kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5\") pod \"7d91c741-89f9-4d75-97e2-e957f7029faf\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " Jul 28 19:45:56.858842 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.858810 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file\") pod \"7d91c741-89f9-4d75-97e2-e957f7029faf\" (UID: \"7d91c741-89f9-4d75-97e2-e957f7029faf\") " Jul 28 19:45:56.859177 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.859147 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file" (OuterVolumeSpecName: "config-file") pod "7d91c741-89f9-4d75-97e2-e957f7029faf" (UID: "7d91c741-89f9-4d75-97e2-e957f7029faf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 19:45:56.860650 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.860626 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5" (OuterVolumeSpecName: "kube-api-access-sktp5") pod "7d91c741-89f9-4d75-97e2-e957f7029faf" (UID: "7d91c741-89f9-4d75-97e2-e957f7029faf"). InnerVolumeSpecName "kube-api-access-sktp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:45:56.960462 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.960428 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d91c741-89f9-4d75-97e2-e957f7029faf-config-file\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:56.960462 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:56.960457 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktp5\" (UniqueName: \"kubernetes.io/projected/7d91c741-89f9-4d75-97e2-e957f7029faf-kube-api-access-sktp5\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:45:57.023403 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.023367 2562 generic.go:356] "Generic (PLEG): container finished" podID="7d91c741-89f9-4d75-97e2-e957f7029faf" containerID="347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758" exitCode=0 Jul 28 19:45:57.023572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.023414 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" Jul 28 19:45:57.023572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.023450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" event={"ID":"7d91c741-89f9-4d75-97e2-e957f7029faf","Type":"ContainerDied","Data":"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758"} Jul 28 19:45:57.023572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.023526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqxjm" event={"ID":"7d91c741-89f9-4d75-97e2-e957f7029faf","Type":"ContainerDied","Data":"023d74fda4c2ef5096d3934628480814081784d0f1c75e2b70fcc99a86377279"} Jul 28 19:45:57.023572 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.023551 2562 scope.go:117] "RemoveContainer" containerID="347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758" Jul 28 19:45:57.032843 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.032827 2562 scope.go:117] "RemoveContainer" containerID="347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758" Jul 28 19:45:57.033095 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:45:57.033077 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758\": container with ID starting with 347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758 not found: ID does not exist" containerID="347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758" Jul 28 19:45:57.033150 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.033109 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758"} err="failed to get container status \"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758\": rpc error: code = NotFound desc = could not find container \"347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758\": container with ID starting with 347a5f517b91578e0836b7bfa02962d4c313e24c8c21488fb1e2521f6d188758 not found: ID does not exist" Jul 28 19:45:57.044696 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.044654 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:57.051914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.051892 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqxjm"] Jul 28 19:45:57.162615 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.162586 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d91c741-89f9-4d75-97e2-e957f7029faf" path="/var/lib/kubelet/pods/7d91c741-89f9-4d75-97e2-e957f7029faf/volumes" Jul 28 19:45:57.905488 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.905458 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn"] Jul 28 19:45:57.905875 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.905827 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d91c741-89f9-4d75-97e2-e957f7029faf" containerName="limitador" Jul 28 19:45:57.905875 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.905838 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d91c741-89f9-4d75-97e2-e957f7029faf" containerName="limitador" Jul 28 19:45:57.905960 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.905910 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d91c741-89f9-4d75-97e2-e957f7029faf" containerName="limitador" Jul 28 19:45:57.910574 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.910556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:57.913068 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.913037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 28 19:45:57.913191 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.913104 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 19:45:57.914130 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.913986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rvd48\"" Jul 28 19:45:57.914130 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.914002 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 19:45:57.917059 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.917034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn"] Jul 28 19:45:57.967656 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.967625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkmg\" (UniqueName: \"kubernetes.io/projected/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-kube-api-access-bkkmg\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:57.967837 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:57.967689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-data\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.068532 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.068499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkmg\" (UniqueName: \"kubernetes.io/projected/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-kube-api-access-bkkmg\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.068532 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.068539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-data\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.068921 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.068903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-data\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.078418 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.078392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkmg\" (UniqueName: \"kubernetes.io/projected/1702c138-b75c-4ba4-9d65-04c9ae73d5b3-kube-api-access-bkkmg\") pod \"postgres-69fb4594cf-8rvkn\" (UID: \"1702c138-b75c-4ba4-9d65-04c9ae73d5b3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.221854 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.221773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:45:58.368003 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:58.367977 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn"] Jul 28 19:45:58.368726 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:45:58.368693 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1702c138_b75c_4ba4_9d65_04c9ae73d5b3.slice/crio-66c26403327c5763b9e8971bba11527d382b16ddbc0b9c49b926569eb688c5b5 WatchSource:0}: Error finding container 66c26403327c5763b9e8971bba11527d382b16ddbc0b9c49b926569eb688c5b5: Status 404 returned error can't find the container with id 66c26403327c5763b9e8971bba11527d382b16ddbc0b9c49b926569eb688c5b5 Jul 28 19:45:59.037017 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:45:59.036985 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" event={"ID":"1702c138-b75c-4ba4-9d65-04c9ae73d5b3","Type":"ContainerStarted","Data":"66c26403327c5763b9e8971bba11527d382b16ddbc0b9c49b926569eb688c5b5"} Jul 28 19:46:04.071072 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:04.071021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" event={"ID":"1702c138-b75c-4ba4-9d65-04c9ae73d5b3","Type":"ContainerStarted","Data":"4fd18f9eb375735ada4a3ff201312770e223b57d556c3d50286cdc1e2ff647a7"} Jul 28 19:46:04.071458 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:04.071190 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:46:04.088705 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:04.088637 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" podStartSLOduration=2.221358462 podStartE2EDuration="7.088623155s" totalImagesPullingTime="4.867264693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:04.08734985 +0000 UTC m=+583.523303802" watchObservedRunningTime="2026-07-28 19:46:04.088623155 +0000 UTC m=+583.524577141" Jul 28 19:46:10.105268 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.105239 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rvkn" Jul 28 19:46:10.682832 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.682795 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:10.686600 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.686580 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:10.710820 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.710789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:10.783712 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.783677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz45\" (UniqueName: \"kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45\") pod \"authorino-5d4b67c446-22jgk\" (UID: \"97a82847-45a4-4221-a809-155eb0c7eb47\") " pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:10.884972 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.884940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz45\" (UniqueName: \"kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45\") pod \"authorino-5d4b67c446-22jgk\" (UID: \"97a82847-45a4-4221-a809-155eb0c7eb47\") " pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:10.893919 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.893890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz45\" (UniqueName: \"kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45\") pod \"authorino-5d4b67c446-22jgk\" (UID: \"97a82847-45a4-4221-a809-155eb0c7eb47\") " pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:10.912515 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.912483 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:10.912769 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.912755 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:10.941911 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.941831 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9d478779-xc6m7"] Jul 28 19:46:10.946601 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.946577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:10.952953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:10.952926 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9d478779-xc6m7"] Jul 28 19:46:11.055210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.055182 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:11.057611 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:11.057567 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a82847_45a4_4221_a809_155eb0c7eb47.slice/crio-7f377b74bfaf66901952a12e7e0d3baf49fd110ba213fea746658cf26c16cc3e WatchSource:0}: Error finding container 7f377b74bfaf66901952a12e7e0d3baf49fd110ba213fea746658cf26c16cc3e: Status 404 returned error can't find the container with id 7f377b74bfaf66901952a12e7e0d3baf49fd110ba213fea746658cf26c16cc3e Jul 28 19:46:11.060134 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.060109 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d9d478779-xc6m7"] Jul 28 19:46:11.060460 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:46:11.060434 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-28bfz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d9d478779-xc6m7" podUID="8e7f9f7c-f317-4f35-a465-9f95925d093a" Jul 28 19:46:11.087040 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.087009 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bfz\" (UniqueName: \"kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz\") pod \"authorino-6d9d478779-xc6m7\" (UID: \"8e7f9f7c-f317-4f35-a465-9f95925d093a\") " pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:11.087148 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.087116 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:46:11.090873 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.090853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.093299 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.093282 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 28 19:46:11.097456 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.097428 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:46:11.113426 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.113394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-22jgk" event={"ID":"97a82847-45a4-4221-a809-155eb0c7eb47","Type":"ContainerStarted","Data":"7f377b74bfaf66901952a12e7e0d3baf49fd110ba213fea746658cf26c16cc3e"} Jul 28 19:46:11.113830 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.113453 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:11.120529 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.120509 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:11.187798 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.187770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.187965 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.187851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28bfz\" (UniqueName: \"kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz\") pod \"authorino-6d9d478779-xc6m7\" (UID: \"8e7f9f7c-f317-4f35-a465-9f95925d093a\") " pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:11.187965 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.187898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6z2\" (UniqueName: \"kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.196806 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.196738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bfz\" (UniqueName: \"kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz\") pod \"authorino-6d9d478779-xc6m7\" (UID: \"8e7f9f7c-f317-4f35-a465-9f95925d093a\") " pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:11.288880 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.288838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28bfz\" (UniqueName: \"kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz\") pod \"8e7f9f7c-f317-4f35-a465-9f95925d093a\" (UID: \"8e7f9f7c-f317-4f35-a465-9f95925d093a\") " Jul 28 19:46:11.289077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.289063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6z2\" (UniqueName: \"kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.289117 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.289098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.290990 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.290965 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz" (OuterVolumeSpecName: "kube-api-access-28bfz") pod "8e7f9f7c-f317-4f35-a465-9f95925d093a" (UID: "8e7f9f7c-f317-4f35-a465-9f95925d093a"). InnerVolumeSpecName "kube-api-access-28bfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:46:11.291372 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.291354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.297075 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.297055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6z2\" (UniqueName: \"kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2\") pod \"authorino-9d5dd7947-9qn8h\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.390290 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.390258 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28bfz\" (UniqueName: \"kubernetes.io/projected/8e7f9f7c-f317-4f35-a465-9f95925d093a-kube-api-access-28bfz\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:46:11.401158 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.401126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:46:11.543357 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:11.543324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:46:11.544897 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:11.544872 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f15f438_de79_4734_82e7_cd867abbcd3a.slice/crio-c895834f0cfa740edf3503d9ba30e0a588b296ee95209ad940af06be1311e799 WatchSource:0}: Error finding container c895834f0cfa740edf3503d9ba30e0a588b296ee95209ad940af06be1311e799: Status 404 returned error can't find the container with id c895834f0cfa740edf3503d9ba30e0a588b296ee95209ad940af06be1311e799 Jul 28 19:46:12.121099 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.121066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-22jgk" event={"ID":"97a82847-45a4-4221-a809-155eb0c7eb47","Type":"ContainerStarted","Data":"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8"} Jul 28 19:46:12.121099 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.121065 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-22jgk" podUID="97a82847-45a4-4221-a809-155eb0c7eb47" containerName="authorino" containerID="cri-o://f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8" gracePeriod=30 Jul 28 19:46:12.122735 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.122704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" event={"ID":"1f15f438-de79-4734-82e7-cd867abbcd3a","Type":"ContainerStarted","Data":"c895834f0cfa740edf3503d9ba30e0a588b296ee95209ad940af06be1311e799"} Jul 28 19:46:12.122834 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.122746 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9d478779-xc6m7" Jul 28 19:46:12.138190 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.138148 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-22jgk" podStartSLOduration=1.792289083 podStartE2EDuration="2.138133925s" totalImagesPullingTime="345.844842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:12.136407003 +0000 UTC m=+591.572360957" watchObservedRunningTime="2026-07-28 19:46:12.138133925 +0000 UTC m=+591.574087878" Jul 28 19:46:12.159642 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.159577 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d9d478779-xc6m7"] Jul 28 19:46:12.163534 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.163513 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d9d478779-xc6m7"] Jul 28 19:46:12.361592 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.361568 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:12.501131 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.501045 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhz45\" (UniqueName: \"kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45\") pod \"97a82847-45a4-4221-a809-155eb0c7eb47\" (UID: \"97a82847-45a4-4221-a809-155eb0c7eb47\") " Jul 28 19:46:12.503080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.503049 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45" (OuterVolumeSpecName: "kube-api-access-dhz45") pod "97a82847-45a4-4221-a809-155eb0c7eb47" (UID: "97a82847-45a4-4221-a809-155eb0c7eb47"). InnerVolumeSpecName "kube-api-access-dhz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:46:12.602034 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:12.601997 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhz45\" (UniqueName: \"kubernetes.io/projected/97a82847-45a4-4221-a809-155eb0c7eb47-kube-api-access-dhz45\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:46:13.130319 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.130283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" event={"ID":"1f15f438-de79-4734-82e7-cd867abbcd3a","Type":"ContainerStarted","Data":"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2"} Jul 28 19:46:13.131536 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.131510 2562 generic.go:356] "Generic (PLEG): container finished" podID="97a82847-45a4-4221-a809-155eb0c7eb47" containerID="f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8" exitCode=0 Jul 28 19:46:13.131634 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.131546 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-22jgk" Jul 28 19:46:13.131634 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.131592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-22jgk" event={"ID":"97a82847-45a4-4221-a809-155eb0c7eb47","Type":"ContainerDied","Data":"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8"} Jul 28 19:46:13.131634 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.131629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-22jgk" event={"ID":"97a82847-45a4-4221-a809-155eb0c7eb47","Type":"ContainerDied","Data":"7f377b74bfaf66901952a12e7e0d3baf49fd110ba213fea746658cf26c16cc3e"} Jul 28 19:46:13.131769 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.131646 2562 scope.go:117] "RemoveContainer" containerID="f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8" Jul 28 19:46:13.140749 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.140732 2562 scope.go:117] "RemoveContainer" containerID="f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8" Jul 28 19:46:13.141020 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:46:13.141003 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8\": container with ID starting with f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8 not found: ID does not exist" containerID="f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8" Jul 28 19:46:13.141070 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.141030 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8"} err="failed to get container status \"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8\": rpc error: code = NotFound desc = could not find container \"f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8\": container with ID starting with f46facda4a2fb6f2becc1407b8822e937d431ed3029abf38024699a5929d76d8 not found: ID does not exist" Jul 28 19:46:13.148222 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.148181 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" podStartSLOduration=1.6304667240000001 podStartE2EDuration="2.148167412s" totalImagesPullingTime="517.700688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:13.1458121 +0000 UTC m=+592.581766052" watchObservedRunningTime="2026-07-28 19:46:13.148167412 +0000 UTC m=+592.584121392" Jul 28 19:46:13.166080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.166045 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7f9f7c-f317-4f35-a465-9f95925d093a" path="/var/lib/kubelet/pods/8e7f9f7c-f317-4f35-a465-9f95925d093a/volumes" Jul 28 19:46:13.166293 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.166275 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:13.168820 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.168794 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-22jgk"] Jul 28 19:46:13.173859 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.173834 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:46:13.174094 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.174044 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-d8kmb" podUID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" containerName="authorino" containerID="cri-o://c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e" gracePeriod=30 Jul 28 19:46:13.432285 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.432222 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:46:13.510812 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.510776 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkxsw\" (UniqueName: \"kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw\") pod \"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114\" (UID: \"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114\") " Jul 28 19:46:13.512806 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.512773 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw" (OuterVolumeSpecName: "kube-api-access-mkxsw") pod "aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" (UID: "aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114"). InnerVolumeSpecName "kube-api-access-mkxsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:46:13.612046 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:13.612010 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxsw\" (UniqueName: \"kubernetes.io/projected/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114-kube-api-access-mkxsw\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:46:14.139208 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.139174 2562 generic.go:356] "Generic (PLEG): container finished" podID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" containerID="c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e" exitCode=0 Jul 28 19:46:14.139650 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.139214 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d8kmb" Jul 28 19:46:14.139650 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.139252 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d8kmb" event={"ID":"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114","Type":"ContainerDied","Data":"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e"} Jul 28 19:46:14.139650 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.139293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d8kmb" event={"ID":"aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114","Type":"ContainerDied","Data":"5790e7ebb44c76fa3d64ab8991b04bfaff23ea47068356d4ce80793bc68ab964"} Jul 28 19:46:14.139650 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.139313 2562 scope.go:117] "RemoveContainer" containerID="c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e" Jul 28 19:46:14.148152 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.148136 2562 scope.go:117] "RemoveContainer" containerID="c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e" Jul 28 19:46:14.148395 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:46:14.148376 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e\": container with ID starting with c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e not found: ID does not exist" containerID="c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e" Jul 28 19:46:14.148457 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.148404 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e"} err="failed to get container status \"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e\": rpc error: code = NotFound desc = could not find container \"c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e\": container with ID starting with c90e3a43404d74fd9fbf0f177171f4670b00174f26f95e6ce79959ca5846115e not found: ID does not exist" Jul 28 19:46:14.160908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.160880 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:46:14.166842 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:14.166818 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d8kmb"] Jul 28 19:46:15.161912 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:15.161872 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a82847-45a4-4221-a809-155eb0c7eb47" path="/var/lib/kubelet/pods/97a82847-45a4-4221-a809-155eb0c7eb47/volumes" Jul 28 19:46:15.162422 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:15.162365 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" path="/var/lib/kubelet/pods/aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114/volumes" Jul 28 19:46:17.899673 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.899634 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:17.900077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900038 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a82847-45a4-4221-a809-155eb0c7eb47" containerName="authorino" Jul 28 19:46:17.900077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900054 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a82847-45a4-4221-a809-155eb0c7eb47" containerName="authorino" Jul 28 19:46:17.900077 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900079 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" containerName="authorino" Jul 28 19:46:17.900194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900085 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" containerName="authorino" Jul 28 19:46:17.900194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900142 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7ee9c3-f6f8-464f-aa59-8fcf9caf4114" containerName="authorino" Jul 28 19:46:17.900194 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.900150 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a82847-45a4-4221-a809-155eb0c7eb47" containerName="authorino" Jul 28 19:46:17.902275 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.902254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:17.906109 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.906089 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 19:46:17.906217 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.906159 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrrdb\"" Jul 28 19:46:17.906691 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.906674 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 19:46:17.919079 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:17.919055 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:18.053042 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.053012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbgz\" (UniqueName: \"kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.053219 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.053093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.088000 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.087964 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:46:18.090566 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.090538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.102210 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.102184 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:46:18.154208 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.154178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbgz\" (UniqueName: \"kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.154412 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.154286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.156811 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.156777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.162722 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.162697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbgz\" (UniqueName: \"kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz\") pod \"maas-controller-58988b5c78-l666r\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.212875 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.212839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:18.255039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.255004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr65v\" (UniqueName: \"kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.255517 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.255437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.355159 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.355129 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:18.356031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.356004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.356159 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:18.356025 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b6039a_f87f_4d6f_b53b_46be027172dc.slice/crio-3ec5518eca09e3789acde69d7f36261b5ede2cc167b3be9b65777a1878d59206 WatchSource:0}: Error finding container 3ec5518eca09e3789acde69d7f36261b5ede2cc167b3be9b65777a1878d59206: Status 404 returned error can't find the container with id 3ec5518eca09e3789acde69d7f36261b5ede2cc167b3be9b65777a1878d59206 Jul 28 19:46:18.356159 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.356062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr65v\" (UniqueName: \"kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.358675 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.358635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.364447 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.364424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr65v\" (UniqueName: \"kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v\") pod \"maas-controller-c5fbf75cf-mfjvt\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.401585 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.401557 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:18.533082 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:18.533057 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:46:18.534297 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:18.534273 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe981df7_2804_452d_aca4_5a1211387ca0.slice/crio-192329a188a8e0b3ed39ea33e8c142b8edb49a4e53a1fd1bb29e2cfead4da054 WatchSource:0}: Error finding container 192329a188a8e0b3ed39ea33e8c142b8edb49a4e53a1fd1bb29e2cfead4da054: Status 404 returned error can't find the container with id 192329a188a8e0b3ed39ea33e8c142b8edb49a4e53a1fd1bb29e2cfead4da054 Jul 28 19:46:19.171892 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:19.171858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58988b5c78-l666r" event={"ID":"26b6039a-f87f-4d6f-b53b-46be027172dc","Type":"ContainerStarted","Data":"3ec5518eca09e3789acde69d7f36261b5ede2cc167b3be9b65777a1878d59206"} Jul 28 19:46:19.176316 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:19.176280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" event={"ID":"fe981df7-2804-452d-aca4-5a1211387ca0","Type":"ContainerStarted","Data":"192329a188a8e0b3ed39ea33e8c142b8edb49a4e53a1fd1bb29e2cfead4da054"} Jul 28 19:46:21.005013 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.004985 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 19:46:21.040993 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.040971 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:46:21.042957 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.042940 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:46:21.193157 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.193116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58988b5c78-l666r" event={"ID":"26b6039a-f87f-4d6f-b53b-46be027172dc","Type":"ContainerStarted","Data":"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff"} Jul 28 19:46:21.193367 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.193193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:21.194953 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.194918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" event={"ID":"fe981df7-2804-452d-aca4-5a1211387ca0","Type":"ContainerStarted","Data":"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69"} Jul 28 19:46:21.195080 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.195020 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:21.216093 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.216038 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58988b5c78-l666r" podStartSLOduration=1.57214507 podStartE2EDuration="4.216023393s" totalImagesPullingTime="2.643878323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:21.213315443 +0000 UTC m=+600.649269395" watchObservedRunningTime="2026-07-28 19:46:21.216023393 +0000 UTC m=+600.651977354" Jul 28 19:46:21.235318 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:21.235269 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" podStartSLOduration=0.761844207 podStartE2EDuration="3.235253086s" totalImagesPullingTime="2.473408879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:21.234466689 +0000 UTC m=+600.670420641" watchObservedRunningTime="2026-07-28 19:46:21.235253086 +0000 UTC m=+600.671207037" Jul 28 19:46:32.203139 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.203108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:46:32.205746 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.203177 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:32.274633 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.274597 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:32.275328 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.275275 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58988b5c78-l666r" podUID="26b6039a-f87f-4d6f-b53b-46be027172dc" containerName="manager" containerID="cri-o://8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff" gracePeriod=30 Jul 28 19:46:32.534345 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.534317 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:32.576220 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.576187 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert\") pod \"26b6039a-f87f-4d6f-b53b-46be027172dc\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " Jul 28 19:46:32.576401 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.576232 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwbgz\" (UniqueName: \"kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz\") pod \"26b6039a-f87f-4d6f-b53b-46be027172dc\" (UID: \"26b6039a-f87f-4d6f-b53b-46be027172dc\") " Jul 28 19:46:32.578373 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.578342 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "26b6039a-f87f-4d6f-b53b-46be027172dc" (UID: "26b6039a-f87f-4d6f-b53b-46be027172dc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:46:32.578525 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.578366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz" (OuterVolumeSpecName: "kube-api-access-dwbgz") pod "26b6039a-f87f-4d6f-b53b-46be027172dc" (UID: "26b6039a-f87f-4d6f-b53b-46be027172dc"). InnerVolumeSpecName "kube-api-access-dwbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:46:32.677961 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.677928 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b6039a-f87f-4d6f-b53b-46be027172dc-webhook-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:46:32.677961 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:32.677958 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbgz\" (UniqueName: \"kubernetes.io/projected/26b6039a-f87f-4d6f-b53b-46be027172dc-kube-api-access-dwbgz\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:46:33.271998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.271966 2562 generic.go:356] "Generic (PLEG): container finished" podID="26b6039a-f87f-4d6f-b53b-46be027172dc" containerID="8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff" exitCode=0 Jul 28 19:46:33.272414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.272011 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58988b5c78-l666r" Jul 28 19:46:33.272414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.272050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58988b5c78-l666r" event={"ID":"26b6039a-f87f-4d6f-b53b-46be027172dc","Type":"ContainerDied","Data":"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff"} Jul 28 19:46:33.272414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.272103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58988b5c78-l666r" event={"ID":"26b6039a-f87f-4d6f-b53b-46be027172dc","Type":"ContainerDied","Data":"3ec5518eca09e3789acde69d7f36261b5ede2cc167b3be9b65777a1878d59206"} Jul 28 19:46:33.272414 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.272125 2562 scope.go:117] "RemoveContainer" containerID="8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff" Jul 28 19:46:33.281383 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.281176 2562 scope.go:117] "RemoveContainer" containerID="8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff" Jul 28 19:46:33.281453 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:46:33.281431 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff\": container with ID starting with 8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff not found: ID does not exist" containerID="8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff" Jul 28 19:46:33.281497 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.281456 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff"} err="failed to get container status \"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff\": rpc error: code = NotFound desc = could not find container \"8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff\": container with ID starting with 8a9c80fedaca53ef5877f051245a510cf1981150ebba0ae9622153bced86d9ff not found: ID does not exist" Jul 28 19:46:33.289686 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.289649 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:33.293970 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:33.293950 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58988b5c78-l666r"] Jul 28 19:46:35.162919 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:35.162888 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b6039a-f87f-4d6f-b53b-46be027172dc" path="/var/lib/kubelet/pods/26b6039a-f87f-4d6f-b53b-46be027172dc/volumes" Jul 28 19:46:51.075389 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.075308 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7968df7757-dn8cc"] Jul 28 19:46:51.075767 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.075757 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b6039a-f87f-4d6f-b53b-46be027172dc" containerName="manager" Jul 28 19:46:51.075811 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.075771 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b6039a-f87f-4d6f-b53b-46be027172dc" containerName="manager" Jul 28 19:46:51.075858 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.075848 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b6039a-f87f-4d6f-b53b-46be027172dc" containerName="manager" Jul 28 19:46:51.081951 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.081933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.085026 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.084834 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 28 19:46:51.085026 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.084957 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\"" Jul 28 19:46:51.087116 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.087090 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7968df7757-dn8cc"] Jul 28 19:46:51.139560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.139529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-maas-api-tls\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.139772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.139594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-openshift-service-ca\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.139772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.139618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkg5\" (UniqueName: \"kubernetes.io/projected/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-kube-api-access-pqkg5\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.240800 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.240769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-openshift-service-ca\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.240800 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.240799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkg5\" (UniqueName: \"kubernetes.io/projected/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-kube-api-access-pqkg5\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.241045 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.240866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-maas-api-tls\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.241517 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.241494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-openshift-service-ca\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.243206 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.243188 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-maas-api-tls\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.251387 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.251359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkg5\" (UniqueName: \"kubernetes.io/projected/5b7a9f43-1f5f-42d5-95f3-167d4d0e2515-kube-api-access-pqkg5\") pod \"maas-api-7968df7757-dn8cc\" (UID: \"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515\") " pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.393923 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.393821 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:51.576391 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.576359 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7968df7757-dn8cc"] Jul 28 19:46:51.578035 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:51.578008 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7a9f43_1f5f_42d5_95f3_167d4d0e2515.slice/crio-477fdbc4d73505c5e51374b60d58f24f424620f0eb21f1c624452d11af817ef8 WatchSource:0}: Error finding container 477fdbc4d73505c5e51374b60d58f24f424620f0eb21f1c624452d11af817ef8: Status 404 returned error can't find the container with id 477fdbc4d73505c5e51374b60d58f24f424620f0eb21f1c624452d11af817ef8 Jul 28 19:46:51.579438 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.579421 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 19:46:51.719092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.719058 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vc598"] Jul 28 19:46:51.724843 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.724819 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.728009 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.727983 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 19:46:51.734965 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.734942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vc598"] Jul 28 19:46:51.826048 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.826013 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hkh7f"] Jul 28 19:46:51.830072 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.830056 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:51.832549 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.832530 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q9v4h\"" Jul 28 19:46:51.838300 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.838280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hkh7f"] Jul 28 19:46:51.847084 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.847056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm55s\" (UniqueName: \"kubernetes.io/projected/b38c912a-eba9-4393-a45d-04ae41664576-kube-api-access-hm55s\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.847202 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.847172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38c912a-eba9-4393-a45d-04ae41664576-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.948153 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.948118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zd88\" (UniqueName: \"kubernetes.io/projected/b8592493-106c-4024-b9df-9a8893a28016-kube-api-access-4zd88\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:51.948153 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.948165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm55s\" (UniqueName: \"kubernetes.io/projected/b38c912a-eba9-4393-a45d-04ae41664576-kube-api-access-hm55s\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.948386 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.948197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8592493-106c-4024-b9df-9a8893a28016-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:51.948386 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.948259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38c912a-eba9-4393-a45d-04ae41664576-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.948930 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.948911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38c912a-eba9-4393-a45d-04ae41664576-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:51.956890 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:51.956863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm55s\" (UniqueName: \"kubernetes.io/projected/b38c912a-eba9-4393-a45d-04ae41664576-kube-api-access-hm55s\") pod \"payload-pre-processing-687dbb6cf6-vc598\" (UID: \"b38c912a-eba9-4393-a45d-04ae41664576\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:52.039641 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.039545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:52.049771 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.049738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8592493-106c-4024-b9df-9a8893a28016-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:52.049901 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.049881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zd88\" (UniqueName: \"kubernetes.io/projected/b8592493-106c-4024-b9df-9a8893a28016-kube-api-access-4zd88\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:52.051060 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.051035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8592493-106c-4024-b9df-9a8893a28016-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:52.058164 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.058137 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zd88\" (UniqueName: \"kubernetes.io/projected/b8592493-106c-4024-b9df-9a8893a28016-kube-api-access-4zd88\") pod \"payload-processing-5cc6cd444b-hkh7f\" (UID: \"b8592493-106c-4024-b9df-9a8893a28016\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:52.143106 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.143075 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:52.181915 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.181719 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vc598"] Jul 28 19:46:52.184359 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:46:52.184328 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38c912a_eba9_4393_a45d_04ae41664576.slice/crio-2be74851f4e9c824440d65bb6a07dfd843da164b43ab2a4f7be7a4519aa8104e WatchSource:0}: Error finding container 2be74851f4e9c824440d65bb6a07dfd843da164b43ab2a4f7be7a4519aa8104e: Status 404 returned error can't find the container with id 2be74851f4e9c824440d65bb6a07dfd843da164b43ab2a4f7be7a4519aa8104e Jul 28 19:46:52.353031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.351894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hkh7f"] Jul 28 19:46:52.386555 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.386519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" event={"ID":"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515","Type":"ContainerStarted","Data":"477fdbc4d73505c5e51374b60d58f24f424620f0eb21f1c624452d11af817ef8"} Jul 28 19:46:52.388640 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.388618 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" event={"ID":"b38c912a-eba9-4393-a45d-04ae41664576","Type":"ContainerStarted","Data":"2be74851f4e9c824440d65bb6a07dfd843da164b43ab2a4f7be7a4519aa8104e"} Jul 28 19:46:52.390192 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:52.390160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" event={"ID":"b8592493-106c-4024-b9df-9a8893a28016","Type":"ContainerStarted","Data":"5df6b3fc7a512d411f852fd757a672ef1d2e18adfd01fdbde48f35cd7aa8e3c1"} Jul 28 19:46:54.411789 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:54.411752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" event={"ID":"5b7a9f43-1f5f-42d5-95f3-167d4d0e2515","Type":"ContainerStarted","Data":"8107bf6789062930ee72e2abf761f2ddca137fa0a591758dbf06b3cd7147056d"} Jul 28 19:46:54.412187 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:54.411845 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:46:54.431487 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:54.431438 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" podStartSLOduration=1.032254021 podStartE2EDuration="3.431424531s" totalImagesPullingTime="2.39917051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:54.428720689 +0000 UTC m=+633.864674667" watchObservedRunningTime="2026-07-28 19:46:54.431424531 +0000 UTC m=+633.867378483" Jul 28 19:46:55.419381 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:55.419342 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" event={"ID":"b38c912a-eba9-4393-a45d-04ae41664576","Type":"ContainerStarted","Data":"8dddbf0d5e96f8ba065640ea5281a49d9f0229d92bd079614d21e63e01cc78e7"} Jul 28 19:46:55.419797 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:55.419709 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:46:55.421175 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:55.421153 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" event={"ID":"b8592493-106c-4024-b9df-9a8893a28016","Type":"ContainerStarted","Data":"632642710e9e233f37f970daa5a09a62149da987a44ac0308ae98b9a5b9b43dd"} Jul 28 19:46:55.436887 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:55.436845 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" podStartSLOduration=1.9344140570000001 podStartE2EDuration="4.43683209s" totalImagesPullingTime="2.502418033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:55.435530045 +0000 UTC m=+634.871484001" watchObservedRunningTime="2026-07-28 19:46:55.43683209 +0000 UTC m=+634.872786043" Jul 28 19:46:56.430378 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:56.430346 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:46:59.448819 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.448791 2562 generic.go:356] "Generic (PLEG): container finished" podID="2282a22c-3b4f-4477-a047-13cd7ebb7d6e" containerID="0f285b66a2617f676c774eeb47da0631904af544ba24c67f232b0141fc17da2f" exitCode=0 Jul 28 19:46:59.449204 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.448864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" event={"ID":"2282a22c-3b4f-4477-a047-13cd7ebb7d6e","Type":"ContainerDied","Data":"0f285b66a2617f676c774eeb47da0631904af544ba24c67f232b0141fc17da2f"} Jul 28 19:46:59.449291 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.449272 2562 scope.go:117] "RemoveContainer" containerID="0f285b66a2617f676c774eeb47da0631904af544ba24c67f232b0141fc17da2f" Jul 28 19:46:59.450186 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.450162 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:46:59.450240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.450228 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:46:59.450282 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.450269 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 19:46:59.474069 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:46:59.474026 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" podStartSLOduration=6.150328105 podStartE2EDuration="8.47401201s" totalImagesPullingTime="2.323683905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:55.454871816 +0000 UTC m=+634.890825771" watchObservedRunningTime="2026-07-28 19:46:59.47401201 +0000 UTC m=+638.909965961" Jul 28 19:47:00.427031 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:00.427003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7968df7757-dn8cc" Jul 28 19:47:00.461642 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:00.460730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" event={"ID":"2282a22c-3b4f-4477-a047-13cd7ebb7d6e","Type":"ContainerStarted","Data":"76fe52aeb91dddcf8ff38c3867a0f74bc43b89cdfd91c89d72573374a082670e"} Jul 28 19:47:00.858839 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:00.858800 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:47:01.456828 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:01.456797 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:47:01.458135 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:01.458103 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-f9dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 19:47:01.458332 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:01.458305 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" podUID="2282a22c-3b4f-4477-a047-13cd7ebb7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 19:47:02.456735 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:02.456700 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-f9dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 19:47:02.457364 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:02.457334 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" podUID="2282a22c-3b4f-4477-a047-13cd7ebb7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 19:47:03.456971 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:03.456928 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-f9dwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 19:47:03.457444 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:03.456992 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" podUID="2282a22c-3b4f-4477-a047-13cd7ebb7d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 19:47:04.460935 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:04.460907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:47:04.461861 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:04.461844 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-f9dwd" Jul 28 19:47:06.432352 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:06.432321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vc598" Jul 28 19:47:07.438442 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:47:07.438410 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hkh7f" Jul 28 19:48:00.188684 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.185203 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt"] Jul 28 19:48:00.197557 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.197526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.203142 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.203113 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hlzv\"" Jul 28 19:48:00.204238 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.204213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 19:48:00.204908 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.204889 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 19:48:00.205610 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.205590 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 28 19:48:00.207395 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.206768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt"] Jul 28 19:48:00.291721 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz8j\" (UniqueName: \"kubernetes.io/projected/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-kube-api-access-zbz8j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.291823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.291823 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.291921 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291860 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.291921 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.292026 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.291949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392393 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz8j\" (UniqueName: \"kubernetes.io/projected/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-kube-api-access-zbz8j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392560 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.392984 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.393070 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.392971 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.393070 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.393013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.394648 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.394630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.394847 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.394828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.400795 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.400776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz8j\" (UniqueName: \"kubernetes.io/projected/a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7-kube-api-access-zbz8j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt\" (UID: \"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.512270 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.512184 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:00.654684 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.654641 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt"] Jul 28 19:48:00.656056 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:48:00.656024 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16044ea_b5f3_4cf6_87f4_8ac8c2929bc7.slice/crio-5dd9ff833168a28baab678fb149bf97412565400adffd9f38b20ae0dafb975e0 WatchSource:0}: Error finding container 5dd9ff833168a28baab678fb149bf97412565400adffd9f38b20ae0dafb975e0: Status 404 returned error can't find the container with id 5dd9ff833168a28baab678fb149bf97412565400adffd9f38b20ae0dafb975e0 Jul 28 19:48:00.806752 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:00.806651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" event={"ID":"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7","Type":"ContainerStarted","Data":"5dd9ff833168a28baab678fb149bf97412565400adffd9f38b20ae0dafb975e0"} Jul 28 19:48:01.813513 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:01.813477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" event={"ID":"a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7","Type":"ContainerStarted","Data":"d6baf995352c2251af759710667e3993c70237889a3820913dfcdf1861a279bc"} Jul 28 19:48:01.833595 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:01.833549 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" podStartSLOduration=0.827279673 podStartE2EDuration="1.833534255s" totalImagesPullingTime="1.006254582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:01.830877637 +0000 UTC m=+701.266831588" watchObservedRunningTime="2026-07-28 19:48:01.833534255 +0000 UTC m=+701.269488207" Jul 28 19:48:10.513060 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:10.513023 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:10.513548 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:10.513091 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:10.525839 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:10.525818 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:10.875455 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:10.875375 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt" Jul 28 19:48:11.602861 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.602823 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2"] Jul 28 19:48:11.608945 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.608919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.609284 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.609261 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2"] Jul 28 19:48:11.611480 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.611463 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 19:48:11.688218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.688218 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688219 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/374aa924-34bb-428f-85d5-5e3872714f67-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.688460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.688460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.688460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbsf\" (UniqueName: \"kubernetes.io/projected/374aa924-34bb-428f-85d5-5e3872714f67-kube-api-access-jmbsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.688460 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.688421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789541 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbsf\" (UniqueName: \"kubernetes.io/projected/374aa924-34bb-428f-85d5-5e3872714f67-kube-api-access-jmbsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789727 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/374aa924-34bb-428f-85d5-5e3872714f67-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.789914 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.789779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.790066 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.790044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.790126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.790081 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.790164 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.790133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.792078 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.792052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/374aa924-34bb-428f-85d5-5e3872714f67-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.792306 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.792288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/374aa924-34bb-428f-85d5-5e3872714f67-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.796674 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.796643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbsf\" (UniqueName: \"kubernetes.io/projected/374aa924-34bb-428f-85d5-5e3872714f67-kube-api-access-jmbsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ppbd2\" (UID: \"374aa924-34bb-428f-85d5-5e3872714f67\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:11.920461 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:11.920430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:12.057630 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:12.057602 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2"] Jul 28 19:48:12.059035 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:48:12.059011 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374aa924_34bb_428f_85d5_5e3872714f67.slice/crio-ebe6865ce163bb3c0b804685266dcd04c72405c3d87a52af2aabd1d62a0dd3f9 WatchSource:0}: Error finding container ebe6865ce163bb3c0b804685266dcd04c72405c3d87a52af2aabd1d62a0dd3f9: Status 404 returned error can't find the container with id ebe6865ce163bb3c0b804685266dcd04c72405c3d87a52af2aabd1d62a0dd3f9 Jul 28 19:48:12.878294 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:12.878261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" event={"ID":"374aa924-34bb-428f-85d5-5e3872714f67","Type":"ContainerStarted","Data":"292080b67c80f25cd86a570c8ff8d8747341dd10b1a936f97c7b5bec86657e0c"} Jul 28 19:48:12.878690 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:12.878316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" event={"ID":"374aa924-34bb-428f-85d5-5e3872714f67","Type":"ContainerStarted","Data":"ebe6865ce163bb3c0b804685266dcd04c72405c3d87a52af2aabd1d62a0dd3f9"} Jul 28 19:48:12.899506 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:12.899460 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" podStartSLOduration=1.721103964 podStartE2EDuration="1.899444787s" totalImagesPullingTime="178.340823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:12.896475113 +0000 UTC m=+712.332429076" watchObservedRunningTime="2026-07-28 19:48:12.899444787 +0000 UTC m=+712.335398739" Jul 28 19:48:21.921313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:21.921227 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:21.921313 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:21.921299 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:21.934048 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:21.934027 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:48:21.948466 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:48:21.948445 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ppbd2" Jul 28 19:49:20.193998 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.193961 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:49:20.194559 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.194198 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" podUID="1f15f438-de79-4734-82e7-cd867abbcd3a" containerName="authorino" containerID="cri-o://c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2" gracePeriod=30 Jul 28 19:49:20.466240 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.466213 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:49:20.576104 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.576067 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert\") pod \"1f15f438-de79-4734-82e7-cd867abbcd3a\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " Jul 28 19:49:20.576293 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.576174 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px6z2\" (UniqueName: \"kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2\") pod \"1f15f438-de79-4734-82e7-cd867abbcd3a\" (UID: \"1f15f438-de79-4734-82e7-cd867abbcd3a\") " Jul 28 19:49:20.578113 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.578085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2" (OuterVolumeSpecName: "kube-api-access-px6z2") pod "1f15f438-de79-4734-82e7-cd867abbcd3a" (UID: "1f15f438-de79-4734-82e7-cd867abbcd3a"). InnerVolumeSpecName "kube-api-access-px6z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:49:20.589009 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.588982 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1f15f438-de79-4734-82e7-cd867abbcd3a" (UID: "1f15f438-de79-4734-82e7-cd867abbcd3a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:49:20.677018 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.676984 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6z2\" (UniqueName: \"kubernetes.io/projected/1f15f438-de79-4734-82e7-cd867abbcd3a-kube-api-access-px6z2\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:49:20.677018 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:20.677011 2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1f15f438-de79-4734-82e7-cd867abbcd3a-tls-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:49:21.276703 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.276653 2562 generic.go:356] "Generic (PLEG): container finished" podID="1f15f438-de79-4734-82e7-cd867abbcd3a" containerID="c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2" exitCode=0 Jul 28 19:49:21.276703 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.276688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" event={"ID":"1f15f438-de79-4734-82e7-cd867abbcd3a","Type":"ContainerDied","Data":"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2"} Jul 28 19:49:21.277132 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.276719 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" Jul 28 19:49:21.277132 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.276731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9d5dd7947-9qn8h" event={"ID":"1f15f438-de79-4734-82e7-cd867abbcd3a","Type":"ContainerDied","Data":"c895834f0cfa740edf3503d9ba30e0a588b296ee95209ad940af06be1311e799"} Jul 28 19:49:21.277132 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.276751 2562 scope.go:117] "RemoveContainer" containerID="c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2" Jul 28 19:49:21.285556 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.285540 2562 scope.go:117] "RemoveContainer" containerID="c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2" Jul 28 19:49:21.285853 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:49:21.285833 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2\": container with ID starting with c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2 not found: ID does not exist" containerID="c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2" Jul 28 19:49:21.285919 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.285866 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2"} err="failed to get container status \"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2\": rpc error: code = NotFound desc = could not find container \"c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2\": container with ID starting with c5b366e77570e9774bf9c8a40609fee167b96da7285c0e49855e1efaa62993f2 not found: ID does not exist" Jul 28 19:49:21.304244 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.304212 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:49:21.311540 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:21.311517 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9d5dd7947-9qn8h"] Jul 28 19:49:23.163583 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:49:23.163549 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f15f438-de79-4734-82e7-cd867abbcd3a" path="/var/lib/kubelet/pods/1f15f438-de79-4734-82e7-cd867abbcd3a/volumes" Jul 28 19:50:08.037431 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.037398 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:50:08.037904 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.037608 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" podUID="fe981df7-2804-452d-aca4-5a1211387ca0" containerName="manager" containerID="cri-o://9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69" gracePeriod=30 Jul 28 19:50:08.296327 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.296268 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:50:08.391983 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.391954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert\") pod \"fe981df7-2804-452d-aca4-5a1211387ca0\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " Jul 28 19:50:08.392181 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.392148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr65v\" (UniqueName: \"kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v\") pod \"fe981df7-2804-452d-aca4-5a1211387ca0\" (UID: \"fe981df7-2804-452d-aca4-5a1211387ca0\") " Jul 28 19:50:08.394035 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.393996 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fe981df7-2804-452d-aca4-5a1211387ca0" (UID: "fe981df7-2804-452d-aca4-5a1211387ca0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:50:08.394154 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.394093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v" (OuterVolumeSpecName: "kube-api-access-cr65v") pod "fe981df7-2804-452d-aca4-5a1211387ca0" (UID: "fe981df7-2804-452d-aca4-5a1211387ca0"). InnerVolumeSpecName "kube-api-access-cr65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:50:08.493752 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.493711 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr65v\" (UniqueName: \"kubernetes.io/projected/fe981df7-2804-452d-aca4-5a1211387ca0-kube-api-access-cr65v\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:50:08.493752 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.493748 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe981df7-2804-452d-aca4-5a1211387ca0-webhook-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:50:08.558629 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.558552 2562 generic.go:356] "Generic (PLEG): container finished" podID="fe981df7-2804-452d-aca4-5a1211387ca0" containerID="9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69" exitCode=0 Jul 28 19:50:08.558629 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.558597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" event={"ID":"fe981df7-2804-452d-aca4-5a1211387ca0","Type":"ContainerDied","Data":"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69"} Jul 28 19:50:08.558629 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.558618 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" Jul 28 19:50:08.558876 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.558641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-mfjvt" event={"ID":"fe981df7-2804-452d-aca4-5a1211387ca0","Type":"ContainerDied","Data":"192329a188a8e0b3ed39ea33e8c142b8edb49a4e53a1fd1bb29e2cfead4da054"} Jul 28 19:50:08.558876 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.558677 2562 scope.go:117] "RemoveContainer" containerID="9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69" Jul 28 19:50:08.568529 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.568507 2562 scope.go:117] "RemoveContainer" containerID="9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69" Jul 28 19:50:08.568818 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:50:08.568798 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69\": container with ID starting with 9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69 not found: ID does not exist" containerID="9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69" Jul 28 19:50:08.568874 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.568826 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69"} err="failed to get container status \"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69\": rpc error: code = NotFound desc = could not find container \"9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69\": container with ID starting with 9a6bd48a194f3e3af8aafd7f796f5c056bce2fee5af02ca6644a5bb9678a6f69 not found: ID does not exist" Jul 28 19:50:08.581098 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.581073 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:50:08.585371 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:08.585345 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-mfjvt"] Jul 28 19:50:09.162652 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:50:09.162618 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe981df7-2804-452d-aca4-5a1211387ca0" path="/var/lib/kubelet/pods/fe981df7-2804-452d-aca4-5a1211387ca0/volumes" Jul 28 19:51:09.422827 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.422791 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:09.423288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423189 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f15f438-de79-4734-82e7-cd867abbcd3a" containerName="authorino" Jul 28 19:51:09.423288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423200 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f15f438-de79-4734-82e7-cd867abbcd3a" containerName="authorino" Jul 28 19:51:09.423288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423214 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe981df7-2804-452d-aca4-5a1211387ca0" containerName="manager" Jul 28 19:51:09.423288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423220 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe981df7-2804-452d-aca4-5a1211387ca0" containerName="manager" Jul 28 19:51:09.423424 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423295 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f15f438-de79-4734-82e7-cd867abbcd3a" containerName="authorino" Jul 28 19:51:09.423424 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.423305 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe981df7-2804-452d-aca4-5a1211387ca0" containerName="manager" Jul 28 19:51:09.426435 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.426418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.429266 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.429234 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrrdb\"" Jul 28 19:51:09.429266 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.429254 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 19:51:09.430251 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.430236 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 19:51:09.442820 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.442792 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:09.550066 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.550033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd62v\" (UniqueName: \"kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.550207 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.550095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.651251 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.651212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.651418 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.651302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd62v\" (UniqueName: \"kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.653772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.653751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.662504 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.662477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd62v\" (UniqueName: \"kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v\") pod \"maas-controller-c5fbf75cf-5tsgm\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.764266 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.764181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:09.909781 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:09.909739 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:09.916205 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:51:09.916178 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7825f99_f8aa_46c4_a979_f86b228ad435.slice/crio-2e9bee4c1e88fcccae44f4ebd16f5b9238fa6718c7330fa3e6042c535c91b885 WatchSource:0}: Error finding container 2e9bee4c1e88fcccae44f4ebd16f5b9238fa6718c7330fa3e6042c535c91b885: Status 404 returned error can't find the container with id 2e9bee4c1e88fcccae44f4ebd16f5b9238fa6718c7330fa3e6042c535c91b885 Jul 28 19:51:10.921075 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:10.921041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" event={"ID":"b7825f99-f8aa-46c4-a979-f86b228ad435","Type":"ContainerStarted","Data":"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1"} Jul 28 19:51:10.921501 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:10.921091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" event={"ID":"b7825f99-f8aa-46c4-a979-f86b228ad435","Type":"ContainerStarted","Data":"2e9bee4c1e88fcccae44f4ebd16f5b9238fa6718c7330fa3e6042c535c91b885"} Jul 28 19:51:10.921501 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:10.921167 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:10.943415 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:10.943365 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" podStartSLOduration=1.472315689 podStartE2EDuration="1.943349825s" totalImagesPullingTime="471.034136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:10.9423078 +0000 UTC m=+890.378261753" watchObservedRunningTime="2026-07-28 19:51:10.943349825 +0000 UTC m=+890.379303800" Jul 28 19:51:21.087340 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:21.087252 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:51:21.088017 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:21.087999 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:51:21.932516 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:21.932488 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:26.595270 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:26.595235 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:26.595693 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:26.595444 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" podUID="b7825f99-f8aa-46c4-a979-f86b228ad435" containerName="manager" containerID="cri-o://8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1" gracePeriod=30 Jul 28 19:51:26.858772 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:26.858710 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:27.013475 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.013445 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert\") pod \"b7825f99-f8aa-46c4-a979-f86b228ad435\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " Jul 28 19:51:27.013693 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.013535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd62v\" (UniqueName: \"kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v\") pod \"b7825f99-f8aa-46c4-a979-f86b228ad435\" (UID: \"b7825f99-f8aa-46c4-a979-f86b228ad435\") " Jul 28 19:51:27.015629 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.015597 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v" (OuterVolumeSpecName: "kube-api-access-sd62v") pod "b7825f99-f8aa-46c4-a979-f86b228ad435" (UID: "b7825f99-f8aa-46c4-a979-f86b228ad435"). InnerVolumeSpecName "kube-api-access-sd62v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:51:27.015931 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.015910 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b7825f99-f8aa-46c4-a979-f86b228ad435" (UID: "b7825f99-f8aa-46c4-a979-f86b228ad435"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 19:51:27.019552 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.019523 2562 generic.go:356] "Generic (PLEG): container finished" podID="b7825f99-f8aa-46c4-a979-f86b228ad435" containerID="8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1" exitCode=0 Jul 28 19:51:27.019704 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.019562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" event={"ID":"b7825f99-f8aa-46c4-a979-f86b228ad435","Type":"ContainerDied","Data":"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1"} Jul 28 19:51:27.019704 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.019578 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" Jul 28 19:51:27.019704 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.019605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-5tsgm" event={"ID":"b7825f99-f8aa-46c4-a979-f86b228ad435","Type":"ContainerDied","Data":"2e9bee4c1e88fcccae44f4ebd16f5b9238fa6718c7330fa3e6042c535c91b885"} Jul 28 19:51:27.019704 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.019627 2562 scope.go:117] "RemoveContainer" containerID="8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1" Jul 28 19:51:27.035461 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.035441 2562 scope.go:117] "RemoveContainer" containerID="8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1" Jul 28 19:51:27.035755 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:51:27.035733 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1\": container with ID starting with 8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1 not found: ID does not exist" containerID="8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1" Jul 28 19:51:27.035800 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.035767 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1"} err="failed to get container status \"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1\": rpc error: code = NotFound desc = could not find container \"8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1\": container with ID starting with 8daa69fb51f35c1d0a399100963f233eab217167d27943e14038523ed54912b1 not found: ID does not exist" Jul 28 19:51:27.047235 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.047211 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:27.050579 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.050556 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-5tsgm"] Jul 28 19:51:27.114957 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.114894 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd62v\" (UniqueName: \"kubernetes.io/projected/b7825f99-f8aa-46c4-a979-f86b228ad435-kube-api-access-sd62v\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:51:27.114957 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.114919 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7825f99-f8aa-46c4-a979-f86b228ad435-webhook-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:51:27.163129 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.163104 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7825f99-f8aa-46c4-a979-f86b228ad435" path="/var/lib/kubelet/pods/b7825f99-f8aa-46c4-a979-f86b228ad435/volumes" Jul 28 19:51:27.840288 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.840251 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-sttpr"] Jul 28 19:51:27.840700 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.840680 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7825f99-f8aa-46c4-a979-f86b228ad435" containerName="manager" Jul 28 19:51:27.840700 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.840693 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7825f99-f8aa-46c4-a979-f86b228ad435" containerName="manager" Jul 28 19:51:27.840790 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.840752 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7825f99-f8aa-46c4-a979-f86b228ad435" containerName="manager" Jul 28 19:51:27.845098 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.845078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:27.847550 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.847526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrrdb\"" Jul 28 19:51:27.847691 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.847581 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 19:51:27.848756 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.848646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 19:51:27.852624 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:27.852590 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-sttpr"] Jul 28 19:51:28.022942 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.022911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8h6\" (UniqueName: \"kubernetes.io/projected/3e097f78-ace7-40d9-8c02-1588c0374336-kube-api-access-tg8h6\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.023168 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.022971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e097f78-ace7-40d9-8c02-1588c0374336-webhook-cert\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.124333 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.124250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8h6\" (UniqueName: \"kubernetes.io/projected/3e097f78-ace7-40d9-8c02-1588c0374336-kube-api-access-tg8h6\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.124478 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.124379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e097f78-ace7-40d9-8c02-1588c0374336-webhook-cert\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.126814 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.126794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e097f78-ace7-40d9-8c02-1588c0374336-webhook-cert\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.133254 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.133230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8h6\" (UniqueName: \"kubernetes.io/projected/3e097f78-ace7-40d9-8c02-1588c0374336-kube-api-access-tg8h6\") pod \"maas-controller-c5fbf75cf-sttpr\" (UID: \"3e097f78-ace7-40d9-8c02-1588c0374336\") " pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.157236 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.157202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:28.298385 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:28.298362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c5fbf75cf-sttpr"] Jul 28 19:51:28.300255 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:51:28.300223 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e097f78_ace7_40d9_8c02_1588c0374336.slice/crio-a041a05de4091d53c052364697dabc5b7511ae198f2333952e00a39e74949c54 WatchSource:0}: Error finding container a041a05de4091d53c052364697dabc5b7511ae198f2333952e00a39e74949c54: Status 404 returned error can't find the container with id a041a05de4091d53c052364697dabc5b7511ae198f2333952e00a39e74949c54 Jul 28 19:51:29.034335 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:29.034301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" event={"ID":"3e097f78-ace7-40d9-8c02-1588c0374336","Type":"ContainerStarted","Data":"18f6b5cbb9d250999696a2aa464b0ad86eb915f97b4101276483b8ab9d11b76d"} Jul 28 19:51:29.034778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:29.034362 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" event={"ID":"3e097f78-ace7-40d9-8c02-1588c0374336","Type":"ContainerStarted","Data":"a041a05de4091d53c052364697dabc5b7511ae198f2333952e00a39e74949c54"} Jul 28 19:51:29.034778 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:29.034507 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:51:29.052126 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:29.052076 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" podStartSLOduration=1.456973359 podStartE2EDuration="2.052061064s" totalImagesPullingTime="595.087705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:29.051649215 +0000 UTC m=+908.487603188" watchObservedRunningTime="2026-07-28 19:51:29.052061064 +0000 UTC m=+908.488015015" Jul 28 19:51:40.043214 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:51:40.043176 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c5fbf75cf-sttpr" Jul 28 19:56:21.127605 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:21.127520 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:56:21.128183 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:21.127615 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 19:56:42.488608 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:42.488573 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:56:42.489159 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:42.488790 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" podUID="537e011a-adf5-43a9-b602-f8024fd02244" containerName="manager" containerID="cri-o://7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24" gracePeriod=10 Jul 28 19:56:45.350423 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.350393 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:56:45.383775 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.383737 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqv7k\" (UniqueName: \"kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k\") pod \"537e011a-adf5-43a9-b602-f8024fd02244\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " Jul 28 19:56:45.383942 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.383786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume\") pod \"537e011a-adf5-43a9-b602-f8024fd02244\" (UID: \"537e011a-adf5-43a9-b602-f8024fd02244\") " Jul 28 19:56:45.384192 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.384170 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "537e011a-adf5-43a9-b602-f8024fd02244" (UID: "537e011a-adf5-43a9-b602-f8024fd02244"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 19:56:45.385845 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.385811 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k" (OuterVolumeSpecName: "kube-api-access-tqv7k") pod "537e011a-adf5-43a9-b602-f8024fd02244" (UID: "537e011a-adf5-43a9-b602-f8024fd02244"). InnerVolumeSpecName "kube-api-access-tqv7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 19:56:45.484416 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.484388 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqv7k\" (UniqueName: \"kubernetes.io/projected/537e011a-adf5-43a9-b602-f8024fd02244-kube-api-access-tqv7k\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:56:45.484416 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.484413 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537e011a-adf5-43a9-b602-f8024fd02244-extensions-socket-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 19:56:45.846849 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.846757 2562 generic.go:356] "Generic (PLEG): container finished" podID="537e011a-adf5-43a9-b602-f8024fd02244" containerID="7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24" exitCode=0 Jul 28 19:56:45.846849 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.846802 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" Jul 28 19:56:45.847110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.846847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" event={"ID":"537e011a-adf5-43a9-b602-f8024fd02244","Type":"ContainerDied","Data":"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24"} Jul 28 19:56:45.847110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.846908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b" event={"ID":"537e011a-adf5-43a9-b602-f8024fd02244","Type":"ContainerDied","Data":"308f8347fa65b354b7396becb2f94b2a7ab1ffb0a2d1a4754d1e93ec1b220c29"} Jul 28 19:56:45.847110 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.846942 2562 scope.go:117] "RemoveContainer" containerID="7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24" Jul 28 19:56:45.857706 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.857691 2562 scope.go:117] "RemoveContainer" containerID="7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24" Jul 28 19:56:45.857947 ip-10-0-132-248 kubenswrapper[2562]: E0728 19:56:45.857926 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24\": container with ID starting with 7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24 not found: ID does not exist" containerID="7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24" Jul 28 19:56:45.858015 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.857959 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24"} err="failed to get container status \"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24\": rpc error: code = NotFound desc = could not find container \"7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24\": container with ID starting with 7ea668812d4cd35868a86441489f87386a0920a4fc191fd272b4eb065dfc1b24 not found: ID does not exist" Jul 28 19:56:45.870039 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.870015 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:56:45.874188 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:45.874169 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhd7b"] Jul 28 19:56:47.162374 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:56:47.162342 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537e011a-adf5-43a9-b602-f8024fd02244" path="/var/lib/kubelet/pods/537e011a-adf5-43a9-b602-f8024fd02244/volumes" Jul 28 19:57:43.644633 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.644594 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s"] Jul 28 19:57:43.645136 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.645013 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537e011a-adf5-43a9-b602-f8024fd02244" containerName="manager" Jul 28 19:57:43.645136 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.645024 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="537e011a-adf5-43a9-b602-f8024fd02244" containerName="manager" Jul 28 19:57:43.645136 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.645092 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="537e011a-adf5-43a9-b602-f8024fd02244" containerName="manager" Jul 28 19:57:43.649345 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.649328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.651939 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.651918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8qgj2\"" Jul 28 19:57:43.668759 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.668736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s"] Jul 28 19:57:43.701092 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.701070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ljq\" (UniqueName: \"kubernetes.io/projected/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-kube-api-access-v4ljq\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.701212 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.701117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.802232 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.802192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ljq\" (UniqueName: \"kubernetes.io/projected/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-kube-api-access-v4ljq\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.802403 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.802250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.802601 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.802584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.813495 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.813468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ljq\" (UniqueName: \"kubernetes.io/projected/5526c3d9-8963-4ed8-85eb-c8012ed42cfe-kube-api-access-v4ljq\") pod \"kuadrant-operator-controller-manager-86878c769f-qtt5s\" (UID: \"5526c3d9-8963-4ed8-85eb-c8012ed42cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:43.960137 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:43.960102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:44.102433 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:44.102400 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s"] Jul 28 19:57:44.104104 ip-10-0-132-248 kubenswrapper[2562]: W0728 19:57:44.104078 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5526c3d9_8963_4ed8_85eb_c8012ed42cfe.slice/crio-91f168b363845455c54905ced916305b5e20c89148c30c839915c9261ba04a0d WatchSource:0}: Error finding container 91f168b363845455c54905ced916305b5e20c89148c30c839915c9261ba04a0d: Status 404 returned error can't find the container with id 91f168b363845455c54905ced916305b5e20c89148c30c839915c9261ba04a0d Jul 28 19:57:44.171345 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:44.171314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" event={"ID":"5526c3d9-8963-4ed8-85eb-c8012ed42cfe","Type":"ContainerStarted","Data":"91f168b363845455c54905ced916305b5e20c89148c30c839915c9261ba04a0d"} Jul 28 19:57:45.178638 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:45.178601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" event={"ID":"5526c3d9-8963-4ed8-85eb-c8012ed42cfe","Type":"ContainerStarted","Data":"d129d90033d5f2f0b06bb6f820318074c8678c1a3b8b4b356dc7edb746d13d7a"} Jul 28 19:57:45.179130 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:45.178696 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 19:57:45.200527 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:45.200471 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" podStartSLOduration=2.200455281 podStartE2EDuration="2.200455281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:57:45.198473618 +0000 UTC m=+1284.634427581" watchObservedRunningTime="2026-07-28 19:57:45.200455281 +0000 UTC m=+1284.636409233" Jul 28 19:57:56.186888 ip-10-0-132-248 kubenswrapper[2562]: I0728 19:57:56.186857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qtt5s" Jul 28 20:00:00.154539 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.154500 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29"] Jul 28 20:00:00.158177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.158142 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" Jul 28 20:00:00.161377 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.161352 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:00:00.168176 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.168155 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" Jul 28 20:00:00.176950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.176926 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29"] Jul 28 20:00:00.524365 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.524335 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29"] Jul 28 20:00:00.526402 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:00:00.526376 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea00239_b3b8_4127_bc9a_37a95dbd1590.slice/crio-3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55 WatchSource:0}: Error finding container 3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55: Status 404 returned error can't find the container with id 3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55 Jul 28 20:00:00.528383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.528366 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:00:00.988726 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:00.988692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" event={"ID":"eea00239-b3b8-4127-bc9a-37a95dbd1590","Type":"ContainerStarted","Data":"3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55"} Jul 28 20:00:19.097562 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:19.097530 2562 generic.go:356] "Generic (PLEG): container finished" podID="eea00239-b3b8-4127-bc9a-37a95dbd1590" containerID="e219041a915cdbcb9deed1c7eeaebd9f5f71e081a434bb28c756745a65dbbfe5" exitCode=0 Jul 28 20:00:19.097951 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:19.097625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" event={"ID":"eea00239-b3b8-4127-bc9a-37a95dbd1590","Type":"ContainerDied","Data":"e219041a915cdbcb9deed1c7eeaebd9f5f71e081a434bb28c756745a65dbbfe5"} Jul 28 20:00:20.243820 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:20.243791 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" Jul 28 20:00:21.110002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:21.109972 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" Jul 28 20:00:21.110002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:21.109992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-nxz29" event={"ID":"eea00239-b3b8-4127-bc9a-37a95dbd1590","Type":"ContainerDied","Data":"3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55"} Jul 28 20:00:21.110219 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:00:21.110058 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea84c3c3e218048f943ea9e9f9703ad9abbb4ba6e7a3c82e794575be7edff55" Jul 28 20:01:21.166615 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:01:21.166581 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:01:21.168859 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:01:21.168839 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:05:15.563156 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.563052 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:15.565345 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.563590 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea00239-b3b8-4127-bc9a-37a95dbd1590" containerName="cleanup" Jul 28 20:05:15.565345 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.563606 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea00239-b3b8-4127-bc9a-37a95dbd1590" containerName="cleanup" Jul 28 20:05:15.565345 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.563717 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea00239-b3b8-4127-bc9a-37a95dbd1590" containerName="cleanup" Jul 28 20:05:15.566821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.566806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.569314 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.569287 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-81375d1e-serving-cert\"" Jul 28 20:05:15.577227 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.577200 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:15.624138 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.624102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2nr\" (UniqueName: \"kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.624341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.624166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.624341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.624278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.724854 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.724819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.725068 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.724901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.725068 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.724951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2nr\" (UniqueName: \"kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.725568 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.725542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.727275 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.727252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.733632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.733599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2nr\" (UniqueName: \"kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr\") pod \"maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:15.879647 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:15.879563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:16.024691 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.024644 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:16.026261 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:05:16.026227 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab14aac7_8bca_4cc0_a37b_8a8cc3d6e17e.slice/crio-02428b1bef92a7ffba6fa77f3ccf595dc954bb239dbf238d03eade56b79625d2 WatchSource:0}: Error finding container 02428b1bef92a7ffba6fa77f3ccf595dc954bb239dbf238d03eade56b79625d2: Status 404 returned error can't find the container with id 02428b1bef92a7ffba6fa77f3ccf595dc954bb239dbf238d03eade56b79625d2 Jul 28 20:05:16.027703 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.027654 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:05:16.270430 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.270396 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:16.275190 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.275169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.278348 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.278328 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-81375d1e\"" Jul 28 20:05:16.278460 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.278350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-81375d1e-dockercfg-bzr9c\"" Jul 28 20:05:16.286025 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.286000 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:16.289480 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.289458 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:16.293690 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.293652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.305282 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.305253 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:16.330103 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.330066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcrs\" (UniqueName: \"kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.330268 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.330113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.330268 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.330157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.330268 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.330211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhbv\" (UniqueName: \"kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.431625 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.431588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcrs\" (UniqueName: \"kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.431851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.431640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.431851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.431737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.431851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.431806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhbv\" (UniqueName: \"kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.432611 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.432588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.432654 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.432588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.440015 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.439964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcrs\" (UniqueName: \"kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs\") pod \"payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.440015 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.439964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhbv\" (UniqueName: \"kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv\") pod \"payload-processing-e2e-ait-81375d1e-5849595856-d5dhr\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.587589 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.587510 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:16.606013 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.605988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:16.832053 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.832019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" event={"ID":"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e","Type":"ContainerStarted","Data":"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede"} Jul 28 20:05:16.832213 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.832075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" event={"ID":"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e","Type":"ContainerStarted","Data":"02428b1bef92a7ffba6fa77f3ccf595dc954bb239dbf238d03eade56b79625d2"} Jul 28 20:05:16.832213 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.832120 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:16.849855 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.849745 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" podStartSLOduration=1.150198345 podStartE2EDuration="1.849722798s" totalImagesPullingTime="699.524453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:16.848192867 +0000 UTC m=+1736.284146845" watchObservedRunningTime="2026-07-28 20:05:16.849722798 +0000 UTC m=+1736.285676751" Jul 28 20:05:16.950656 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.950629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:16.953290 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:05:16.953263 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb0359f_76b0_4c33_88d9_361b4b71d19c.slice/crio-b8cafd5c5cb2fc00dd6fc5decaad248633adbdd69e1c64990eafdb757c9c04a2 WatchSource:0}: Error finding container b8cafd5c5cb2fc00dd6fc5decaad248633adbdd69e1c64990eafdb757c9c04a2: Status 404 returned error can't find the container with id b8cafd5c5cb2fc00dd6fc5decaad248633adbdd69e1c64990eafdb757c9c04a2 Jul 28 20:05:16.991220 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:16.991191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:16.991961 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:05:16.991932 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76581862_81d3_4e16_9652_2d5e13e8f6f7.slice/crio-c6c709f3193b6326ade41f1f076fb4c54f01cc2f18c8dc5cca94cf411d9269e6 WatchSource:0}: Error finding container c6c709f3193b6326ade41f1f076fb4c54f01cc2f18c8dc5cca94cf411d9269e6: Status 404 returned error can't find the container with id c6c709f3193b6326ade41f1f076fb4c54f01cc2f18c8dc5cca94cf411d9269e6 Jul 28 20:05:17.839078 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.839033 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" event={"ID":"76581862-81d3-4e16-9652-2d5e13e8f6f7","Type":"ContainerStarted","Data":"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96"} Jul 28 20:05:17.839556 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.839175 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:17.839556 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.839204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" event={"ID":"76581862-81d3-4e16-9652-2d5e13e8f6f7","Type":"ContainerStarted","Data":"c6c709f3193b6326ade41f1f076fb4c54f01cc2f18c8dc5cca94cf411d9269e6"} Jul 28 20:05:17.840890 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.840858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" event={"ID":"aeb0359f-76b0-4c33-88d9-361b4b71d19c","Type":"ContainerStarted","Data":"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de"} Jul 28 20:05:17.841038 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.840915 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" event={"ID":"aeb0359f-76b0-4c33-88d9-361b4b71d19c","Type":"ContainerStarted","Data":"b8cafd5c5cb2fc00dd6fc5decaad248633adbdd69e1c64990eafdb757c9c04a2"} Jul 28 20:05:17.841038 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.840964 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:17.856882 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.856834 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" podStartSLOduration=1.85682097 podStartE2EDuration="1.85682097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:17.854313362 +0000 UTC m=+1737.290267312" watchObservedRunningTime="2026-07-28 20:05:17.85682097 +0000 UTC m=+1737.292774945" Jul 28 20:05:17.869159 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:17.869102 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" podStartSLOduration=1.869084129 podStartE2EDuration="1.869084129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:17.868636262 +0000 UTC m=+1737.304590216" watchObservedRunningTime="2026-07-28 20:05:17.869084129 +0000 UTC m=+1737.305038082" Jul 28 20:05:22.847562 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:22.847525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:28.850162 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:28.850130 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:28.850583 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:28.850208 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:41.076863 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.076830 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn"] Jul 28 20:05:41.080767 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.080749 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:41.084069 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.084012 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:05:41.087766 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.087739 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn"] Jul 28 20:05:41.171944 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.171906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81375d1e-srhvn\" (UID: \"834e1727-b1b7-42ab-9884-d80a88edaf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:41.272981 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.272943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81375d1e-srhvn\" (UID: \"834e1727-b1b7-42ab-9884-d80a88edaf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:41.273690 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.273650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81375d1e-srhvn\" (UID: \"834e1727-b1b7-42ab-9884-d80a88edaf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:41.395151 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.395050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:41.543106 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.543080 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn"] Jul 28 20:05:41.544871 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:05:41.544841 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834e1727_b1b7_42ab_9884_d80a88edaf9d.slice/crio-b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d WatchSource:0}: Error finding container b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d: Status 404 returned error can't find the container with id b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d Jul 28 20:05:41.995111 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.995076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" event={"ID":"834e1727-b1b7-42ab-9884-d80a88edaf9d","Type":"ContainerStarted","Data":"5c23b104646bff6dacfa598a7f88f8cf82180fbc476cf3019372f60567587fee"} Jul 28 20:05:41.995296 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:41.995166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" event={"ID":"834e1727-b1b7-42ab-9884-d80a88edaf9d","Type":"ContainerStarted","Data":"b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d"} Jul 28 20:05:42.013347 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:42.013231 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" podStartSLOduration=1.013214547 podStartE2EDuration="1.013214547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:42.011728907 +0000 UTC m=+1761.447682860" watchObservedRunningTime="2026-07-28 20:05:42.013214547 +0000 UTC m=+1761.449168499" Jul 28 20:05:43.003113 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:43.003083 2562 generic.go:356] "Generic (PLEG): container finished" podID="834e1727-b1b7-42ab-9884-d80a88edaf9d" containerID="5c23b104646bff6dacfa598a7f88f8cf82180fbc476cf3019372f60567587fee" exitCode=0 Jul 28 20:05:43.003511 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:43.003174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" event={"ID":"834e1727-b1b7-42ab-9884-d80a88edaf9d","Type":"ContainerDied","Data":"5c23b104646bff6dacfa598a7f88f8cf82180fbc476cf3019372f60567587fee"} Jul 28 20:05:44.153308 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:44.153277 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:44.300366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:44.300281 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca\") pod \"834e1727-b1b7-42ab-9884-d80a88edaf9d\" (UID: \"834e1727-b1b7-42ab-9884-d80a88edaf9d\") " Jul 28 20:05:44.300742 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:44.300713 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "834e1727-b1b7-42ab-9884-d80a88edaf9d" (UID: "834e1727-b1b7-42ab-9884-d80a88edaf9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:05:44.401316 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:44.401275 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834e1727-b1b7-42ab-9884-d80a88edaf9d-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:45.016794 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:45.016763 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" Jul 28 20:05:45.016992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:45.016808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn" event={"ID":"834e1727-b1b7-42ab-9884-d80a88edaf9d","Type":"ContainerDied","Data":"b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d"} Jul 28 20:05:45.016992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:45.016865 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b15d9d081b97d9bf0ca5954bf6ef2b1250fac4692d24197832130b7fc9fa147d" Jul 28 20:05:46.153317 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.153266 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:46.153841 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.153516 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" podUID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" containerName="maas-api" containerID="cri-o://6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede" gracePeriod=30 Jul 28 20:05:46.206611 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.206571 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:46.207210 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.206881 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" podUID="76581862-81d3-4e16-9652-2d5e13e8f6f7" containerName="payload-processing" containerID="cri-o://bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96" gracePeriod=30 Jul 28 20:05:46.215560 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.215530 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:46.216213 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.216136 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" podUID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" containerName="payload-pre-processing" containerID="cri-o://e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de" gracePeriod=30 Jul 28 20:05:46.354416 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.354351 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn"] Jul 28 20:05:46.357699 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.357643 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81375d1e-srhvn"] Jul 28 20:05:46.511976 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.511949 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:46.548506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.548476 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:46.551024 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.550999 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:46.630242 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.630207 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca\") pod \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " Jul 28 20:05:46.630419 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.630252 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls\") pod \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " Jul 28 20:05:46.630419 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.630378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq2nr\" (UniqueName: \"kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr\") pod \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\" (UID: \"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e\") " Jul 28 20:05:46.630718 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.630649 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" (UID: "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:05:46.632478 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.632454 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" (UID: "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:05:46.632478 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.632462 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr" (OuterVolumeSpecName: "kube-api-access-dq2nr") pod "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" (UID: "ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e"). InnerVolumeSpecName "kube-api-access-dq2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:05:46.731698 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731580 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwcrs\" (UniqueName: \"kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs\") pod \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " Jul 28 20:05:46.731698 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731635 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume\") pod \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\" (UID: \"aeb0359f-76b0-4c33-88d9-361b4b71d19c\") " Jul 28 20:05:46.731902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hhbv\" (UniqueName: \"kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv\") pod \"76581862-81d3-4e16-9652-2d5e13e8f6f7\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " Jul 28 20:05:46.731902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731774 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume\") pod \"76581862-81d3-4e16-9652-2d5e13e8f6f7\" (UID: \"76581862-81d3-4e16-9652-2d5e13e8f6f7\") " Jul 28 20:05:46.731975 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731962 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2nr\" (UniqueName: \"kubernetes.io/projected/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-kube-api-access-dq2nr\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.732010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731974 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-openshift-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.732010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.731984 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e-maas-api-tls\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.732110 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.732074 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aeb0359f-76b0-4c33-88d9-361b4b71d19c" (UID: "aeb0359f-76b0-4c33-88d9-361b4b71d19c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:05:46.732229 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.732165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76581862-81d3-4e16-9652-2d5e13e8f6f7" (UID: "76581862-81d3-4e16-9652-2d5e13e8f6f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:05:46.733839 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.733815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs" (OuterVolumeSpecName: "kube-api-access-nwcrs") pod "aeb0359f-76b0-4c33-88d9-361b4b71d19c" (UID: "aeb0359f-76b0-4c33-88d9-361b4b71d19c"). InnerVolumeSpecName "kube-api-access-nwcrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:05:46.733922 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.733860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv" (OuterVolumeSpecName: "kube-api-access-8hhbv") pod "76581862-81d3-4e16-9652-2d5e13e8f6f7" (UID: "76581862-81d3-4e16-9652-2d5e13e8f6f7"). InnerVolumeSpecName "kube-api-access-8hhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:05:46.832902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.832860 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76581862-81d3-4e16-9652-2d5e13e8f6f7-plugins-config-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.832902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.832896 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcrs\" (UniqueName: \"kubernetes.io/projected/aeb0359f-76b0-4c33-88d9-361b4b71d19c-kube-api-access-nwcrs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.832902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.832905 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb0359f-76b0-4c33-88d9-361b4b71d19c-plugins-config-volume\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:46.833146 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:46.832917 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhbv\" (UniqueName: \"kubernetes.io/projected/76581862-81d3-4e16-9652-2d5e13e8f6f7-kube-api-access-8hhbv\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:05:47.030192 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.030103 2562 generic.go:356] "Generic (PLEG): container finished" podID="76581862-81d3-4e16-9652-2d5e13e8f6f7" containerID="bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96" exitCode=0 Jul 28 20:05:47.030192 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.030149 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" Jul 28 20:05:47.030402 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.030190 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" event={"ID":"76581862-81d3-4e16-9652-2d5e13e8f6f7","Type":"ContainerDied","Data":"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96"} Jul 28 20:05:47.030402 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.030284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr" event={"ID":"76581862-81d3-4e16-9652-2d5e13e8f6f7","Type":"ContainerDied","Data":"c6c709f3193b6326ade41f1f076fb4c54f01cc2f18c8dc5cca94cf411d9269e6"} Jul 28 20:05:47.030402 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.030325 2562 scope.go:117] "RemoveContainer" containerID="bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96" Jul 28 20:05:47.031549 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.031529 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" containerID="6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede" exitCode=0 Jul 28 20:05:47.031656 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.031572 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" Jul 28 20:05:47.031656 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.031609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" event={"ID":"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e","Type":"ContainerDied","Data":"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede"} Jul 28 20:05:47.031656 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.031649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm" event={"ID":"ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e","Type":"ContainerDied","Data":"02428b1bef92a7ffba6fa77f3ccf595dc954bb239dbf238d03eade56b79625d2"} Jul 28 20:05:47.033194 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.033175 2562 generic.go:356] "Generic (PLEG): container finished" podID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" containerID="e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de" exitCode=0 Jul 28 20:05:47.033304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.033224 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" Jul 28 20:05:47.033304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.033268 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" event={"ID":"aeb0359f-76b0-4c33-88d9-361b4b71d19c","Type":"ContainerDied","Data":"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de"} Jul 28 20:05:47.033304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.033298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r" event={"ID":"aeb0359f-76b0-4c33-88d9-361b4b71d19c","Type":"ContainerDied","Data":"b8cafd5c5cb2fc00dd6fc5decaad248633adbdd69e1c64990eafdb757c9c04a2"} Jul 28 20:05:47.040812 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.040757 2562 scope.go:117] "RemoveContainer" containerID="bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96" Jul 28 20:05:47.041231 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:05:47.041087 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96\": container with ID starting with bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96 not found: ID does not exist" containerID="bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96" Jul 28 20:05:47.041231 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.041123 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96"} err="failed to get container status \"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96\": rpc error: code = NotFound desc = could not find container \"bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96\": container with ID starting with bdd87132883e3e3c974d4c89c7926a978a6f818e8404b6810ba5bd83787ddf96 not found: ID does not exist" Jul 28 20:05:47.041231 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.041145 2562 scope.go:117] "RemoveContainer" containerID="6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede" Jul 28 20:05:47.052253 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.052233 2562 scope.go:117] "RemoveContainer" containerID="6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede" Jul 28 20:05:47.052553 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:05:47.052533 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede\": container with ID starting with 6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede not found: ID does not exist" containerID="6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede" Jul 28 20:05:47.052621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.052559 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede"} err="failed to get container status \"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede\": rpc error: code = NotFound desc = could not find container \"6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede\": container with ID starting with 6c758b5976ee3f8dc1f5f56980d38ebef27999ea909aa778312ff69597702ede not found: ID does not exist" Jul 28 20:05:47.052621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.052575 2562 scope.go:117] "RemoveContainer" containerID="e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de" Jul 28 20:05:47.061529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.061504 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:47.062028 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.062011 2562 scope.go:117] "RemoveContainer" containerID="e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de" Jul 28 20:05:47.062342 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:05:47.062318 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de\": container with ID starting with e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de not found: ID does not exist" containerID="e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de" Jul 28 20:05:47.062424 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.062347 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de"} err="failed to get container status \"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de\": rpc error: code = NotFound desc = could not find container \"e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de\": container with ID starting with e07cddb5ec1c53c75cc0b1446adebfa761f9f68c446c645ca9ae37c621b9b8de not found: ID does not exist" Jul 28 20:05:47.066879 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.066856 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81375d1e-7dbfbcbfdd-hdhjm"] Jul 28 20:05:47.081374 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.081346 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:47.088420 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.088393 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-81375d1e-5849595856-d5dhr"] Jul 28 20:05:47.105547 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.105516 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:47.111469 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.111438 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-81375d1e-784b8dcf6d-rpm9r"] Jul 28 20:05:47.163954 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.163923 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76581862-81d3-4e16-9652-2d5e13e8f6f7" path="/var/lib/kubelet/pods/76581862-81d3-4e16-9652-2d5e13e8f6f7/volumes" Jul 28 20:05:47.164418 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.164401 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834e1727-b1b7-42ab-9884-d80a88edaf9d" path="/var/lib/kubelet/pods/834e1727-b1b7-42ab-9884-d80a88edaf9d/volumes" Jul 28 20:05:47.164957 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.164941 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" path="/var/lib/kubelet/pods/ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e/volumes" Jul 28 20:05:47.165403 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.165387 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" path="/var/lib/kubelet/pods/aeb0359f-76b0-4c33-88d9-361b4b71d19c/volumes" Jul 28 20:05:47.468467 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.468420 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:05:47.469203 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469165 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834e1727-b1b7-42ab-9884-d80a88edaf9d" containerName="revoke-keys" Jul 28 20:05:47.469203 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469193 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="834e1727-b1b7-42ab-9884-d80a88edaf9d" containerName="revoke-keys" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469217 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" containerName="maas-api" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469223 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" containerName="maas-api" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469229 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" containerName="payload-pre-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469235 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" containerName="payload-pre-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469253 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76581862-81d3-4e16-9652-2d5e13e8f6f7" containerName="payload-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469263 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76581862-81d3-4e16-9652-2d5e13e8f6f7" containerName="payload-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469354 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb0359f-76b0-4c33-88d9-361b4b71d19c" containerName="payload-pre-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469363 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76581862-81d3-4e16-9652-2d5e13e8f6f7" containerName="payload-processing" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469369 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab14aac7-8bca-4cc0-a37b-8a8cc3d6e17e" containerName="maas-api" Jul 28 20:05:47.469413 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.469377 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="834e1727-b1b7-42ab-9884-d80a88edaf9d" containerName="revoke-keys" Jul 28 20:05:47.474530 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.474508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.477220 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.477201 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bdcbbca7-gw-openshift-default-dockercfg-t7dvg\"" Jul 28 20:05:47.487727 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.487153 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:05:47.640254 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640470 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640470 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640470 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7mn\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.640916 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.640650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.741908 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.741804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.741908 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.741871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.741908 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.741893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.741908 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.741910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742111 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7mn\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.742705 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.742536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.743182 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.743158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.744640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.744618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.745310 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.745289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.770817 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.770785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.770964 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.770839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7mn\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn\") pod \"e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.785795 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.785762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:47.941950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.941918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:05:47.945424 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:05:47.945389 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c345ee_ad91_49f1_b63e_85c493571e22.slice/crio-6b51820f79aa8c9473446ad81a314fb251a85d4e89139d2b0bf59bee89e0da82 WatchSource:0}: Error finding container 6b51820f79aa8c9473446ad81a314fb251a85d4e89139d2b0bf59bee89e0da82: Status 404 returned error can't find the container with id 6b51820f79aa8c9473446ad81a314fb251a85d4e89139d2b0bf59bee89e0da82 Jul 28 20:05:47.947614 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.947581 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:05:47.947739 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.947647 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:05:47.947739 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:47.947693 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:05:48.046286 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:48.046037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerStarted","Data":"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5"} Jul 28 20:05:48.046286 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:48.046106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerStarted","Data":"6b51820f79aa8c9473446ad81a314fb251a85d4e89139d2b0bf59bee89e0da82"} Jul 28 20:05:48.068580 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:48.068523 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podStartSLOduration=1.0685057040000001 podStartE2EDuration="1.068505704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:48.067398676 +0000 UTC m=+1767.503352641" watchObservedRunningTime="2026-07-28 20:05:48.068505704 +0000 UTC m=+1767.504459655" Jul 28 20:05:48.785926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:48.785888 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:49.634249 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.634209 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:05:49.637977 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.637952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.640554 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.640528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bdcbbca7-serving-cert\"" Jul 28 20:05:49.649978 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.649816 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:05:49.762396 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.762358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.762571 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.762435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94zq\" (UniqueName: \"kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.762571 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.762524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.791532 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.791504 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:49.863708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.863622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.863910 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.863794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94zq\" (UniqueName: \"kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.863910 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.863831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.864508 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.864483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.866152 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.866127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.883208 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.883174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94zq\" (UniqueName: \"kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq\") pod \"maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:49.952514 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:49.952480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:50.062002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:50.061970 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:50.063090 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:50.063067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:05:50.108200 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:50.108158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:05:51.069623 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:51.069589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" event={"ID":"7f3e917c-1c6f-4581-8072-b2eb2df4b56d","Type":"ContainerStarted","Data":"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb"} Jul 28 20:05:51.070124 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:51.069645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" event={"ID":"7f3e917c-1c6f-4581-8072-b2eb2df4b56d","Type":"ContainerStarted","Data":"d2aaec793d750221c85b2fde389b082e1e4195b67f02359c4639c9d0f0796113"} Jul 28 20:05:51.070124 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:51.069877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:05:51.085590 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:51.085539 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" podStartSLOduration=1.7589464320000001 podStartE2EDuration="2.08552519s" totalImagesPullingTime="326.578758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:51.084479791 +0000 UTC m=+1770.520433774" watchObservedRunningTime="2026-07-28 20:05:51.08552519 +0000 UTC m=+1770.521479141" Jul 28 20:05:57.080843 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:05:57.080812 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:06:05.063354 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.063268 2562 patch_prober.go:28] interesting pod/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 20:06:05.063354 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.063336 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 20:06:05.156376 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.156344 2562 generic.go:356] "Generic (PLEG): container finished" podID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerID="6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5" exitCode=0 Jul 28 20:06:05.161949 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.161920 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerDied","Data":"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5"} Jul 28 20:06:05.162277 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.162262 2562 scope.go:117] "RemoveContainer" containerID="6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5" Jul 28 20:06:05.163269 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.163245 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:05.163322 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.163307 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:05.163369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:05.163346 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:06.173122 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:06.173076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerStarted","Data":"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19"} Jul 28 20:06:06.785961 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:06.785914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:06.786231 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:06.786204 2562 patch_prober.go:28] interesting pod/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 20:06:06.786317 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:06.786259 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 20:06:07.786730 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:07.786692 2562 patch_prober.go:28] interesting pod/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 20:06:07.787140 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:07.786765 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 20:06:08.787231 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:08.787192 2562 patch_prober.go:28] interesting pod/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 20:06:08.787630 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:08.787255 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 20:06:09.787011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:09.786973 2562 patch_prober.go:28] interesting pod/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 20:06:09.787221 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:09.787031 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 20:06:10.791278 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:10.791240 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:10.791790 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:10.791769 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:10.793000 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:10.792981 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:21.205959 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:21.205930 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:06:21.211623 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:21.211595 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:06:30.236846 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.236813 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:06:30.237333 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.237085 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" podUID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" containerName="maas-api" containerID="cri-o://f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb" gracePeriod=30 Jul 28 20:06:30.513959 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.513929 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:06:30.567412 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.567378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls\") pod \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " Jul 28 20:06:30.567626 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.567478 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca\") pod \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " Jul 28 20:06:30.567626 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.567531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l94zq\" (UniqueName: \"kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq\") pod \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\" (UID: \"7f3e917c-1c6f-4581-8072-b2eb2df4b56d\") " Jul 28 20:06:30.567943 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.567899 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f3e917c-1c6f-4581-8072-b2eb2df4b56d" (UID: "7f3e917c-1c6f-4581-8072-b2eb2df4b56d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:06:30.571461 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.571412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq" (OuterVolumeSpecName: "kube-api-access-l94zq") pod "7f3e917c-1c6f-4581-8072-b2eb2df4b56d" (UID: "7f3e917c-1c6f-4581-8072-b2eb2df4b56d"). InnerVolumeSpecName "kube-api-access-l94zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:06:30.572305 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.572278 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f3e917c-1c6f-4581-8072-b2eb2df4b56d" (UID: "7f3e917c-1c6f-4581-8072-b2eb2df4b56d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:06:30.574389 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.574363 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:06:30.574740 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.574710 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" containerID="cri-o://fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19" gracePeriod=30 Jul 28 20:06:30.668522 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.668476 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-maas-api-tls\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:30.668522 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.668513 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-openshift-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:30.668522 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:30.668524 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94zq\" (UniqueName: \"kubernetes.io/projected/7f3e917c-1c6f-4581-8072-b2eb2df4b56d-kube-api-access-l94zq\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:31.324572 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.324534 2562 generic.go:356] "Generic (PLEG): container finished" podID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" containerID="f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb" exitCode=0 Jul 28 20:06:31.325034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.324576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" event={"ID":"7f3e917c-1c6f-4581-8072-b2eb2df4b56d","Type":"ContainerDied","Data":"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb"} Jul 28 20:06:31.325034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.324599 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" Jul 28 20:06:31.325034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.324646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl" event={"ID":"7f3e917c-1c6f-4581-8072-b2eb2df4b56d","Type":"ContainerDied","Data":"d2aaec793d750221c85b2fde389b082e1e4195b67f02359c4639c9d0f0796113"} Jul 28 20:06:31.325034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.324683 2562 scope.go:117] "RemoveContainer" containerID="f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb" Jul 28 20:06:31.335295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.335266 2562 scope.go:117] "RemoveContainer" containerID="f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb" Jul 28 20:06:31.335651 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:06:31.335624 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb\": container with ID starting with f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb not found: ID does not exist" containerID="f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb" Jul 28 20:06:31.335766 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.335685 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb"} err="failed to get container status \"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb\": rpc error: code = NotFound desc = could not find container \"f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb\": container with ID starting with f245420be7c9860f9519a18473e745e097eaf2680ed65b2037bc4848e8cd96cb not found: ID does not exist" Jul 28 20:06:31.351161 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.351129 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:06:31.361256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:31.361219 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcbbca7-55fb8f7857-jzjbl"] Jul 28 20:06:33.163636 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:33.163598 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" path="/var/lib/kubelet/pods/7f3e917c-1c6f-4581-8072-b2eb2df4b56d/volumes" Jul 28 20:06:35.823195 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.823169 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:35.915450 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915418 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915465 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd7mn\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915513 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915566 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915610 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915689 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915718 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915746 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs\") pod \"f0c345ee-ad91-49f1-b63e-85c493571e22\" (UID: \"f0c345ee-ad91-49f1-b63e-85c493571e22\") " Jul 28 20:06:35.915921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:06:35.916122 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.915929 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:06:35.916122 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.916102 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:06:35.916227 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.916140 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-credential-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:35.916227 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.916156 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:35.916520 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.916486 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:06:35.916673 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.916633 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:06:35.917888 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.917860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:06:35.917970 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.917915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn" (OuterVolumeSpecName: "kube-api-access-hd7mn") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "kube-api-access-hd7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:06:35.918191 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.918172 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:06:35.918230 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:35.918190 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0c345ee-ad91-49f1-b63e-85c493571e22" (UID: "f0c345ee-ad91-49f1-b63e-85c493571e22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:06:36.017263 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017225 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c345ee-ad91-49f1-b63e-85c493571e22-istiod-ca-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017263 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017259 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017263 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017271 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-envoy\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017282 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-workload-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017293 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-podinfo\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017303 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd7mn\" (UniqueName: \"kubernetes.io/projected/f0c345ee-ad91-49f1-b63e-85c493571e22-kube-api-access-hd7mn\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.017506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.017315 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c345ee-ad91-49f1-b63e-85c493571e22-istio-data\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:06:36.367154 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.367076 2562 generic.go:356] "Generic (PLEG): container finished" podID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerID="fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19" exitCode=0 Jul 28 20:06:36.367154 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.367132 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" Jul 28 20:06:36.367332 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.367159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerDied","Data":"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19"} Jul 28 20:06:36.367332 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.367231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x" event={"ID":"f0c345ee-ad91-49f1-b63e-85c493571e22","Type":"ContainerDied","Data":"6b51820f79aa8c9473446ad81a314fb251a85d4e89139d2b0bf59bee89e0da82"} Jul 28 20:06:36.367332 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.367253 2562 scope.go:117] "RemoveContainer" containerID="fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19" Jul 28 20:06:36.380717 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.380696 2562 scope.go:117] "RemoveContainer" containerID="6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5" Jul 28 20:06:36.389601 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.389575 2562 scope.go:117] "RemoveContainer" containerID="fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19" Jul 28 20:06:36.389883 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:06:36.389860 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19\": container with ID starting with fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19 not found: ID does not exist" containerID="fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19" Jul 28 20:06:36.389978 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.389890 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19"} err="failed to get container status \"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19\": rpc error: code = NotFound desc = could not find container \"fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19\": container with ID starting with fd7ce521609e07b25ef6282e9c1eb3276176aacf04b673e3eea956782a348e19 not found: ID does not exist" Jul 28 20:06:36.389978 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.389905 2562 scope.go:117] "RemoveContainer" containerID="6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5" Jul 28 20:06:36.390135 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:06:36.390120 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5\": container with ID starting with 6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5 not found: ID does not exist" containerID="6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5" Jul 28 20:06:36.390180 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.390137 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5"} err="failed to get container status \"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5\": rpc error: code = NotFound desc = could not find container \"6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5\": container with ID starting with 6178ecfb793a56ead63428c5c49934f03c67e63567ab45443c58ffd9163a88d5 not found: ID does not exist" Jul 28 20:06:36.395139 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.395115 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:06:36.398862 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:36.398840 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bdcbbca7-gw-openshift-default-5687bd59cb-s457x"] Jul 28 20:06:37.163381 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:37.163340 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" path="/var/lib/kubelet/pods/f0c345ee-ad91-49f1-b63e-85c493571e22/volumes" Jul 28 20:06:59.642077 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642030 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd"] Jul 28 20:06:59.642750 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642729 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" containerName="maas-api" Jul 28 20:06:59.642840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642752 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" containerName="maas-api" Jul 28 20:06:59.642840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642776 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:06:59.642840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642785 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:06:59.642840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642798 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:06:59.642840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642807 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:06:59.643083 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642904 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:06:59.643083 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.642918 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3e917c-1c6f-4581-8072-b2eb2df4b56d" containerName="maas-api" Jul 28 20:06:59.646165 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.646142 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:06:59.649269 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.649246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:06:59.653268 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.653242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd"] Jul 28 20:06:59.745295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.745251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd\" (UID: \"4fa161cc-0cda-408d-9ecd-15d8535105c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:06:59.846535 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.846495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd\" (UID: \"4fa161cc-0cda-408d-9ecd-15d8535105c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:06:59.847189 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.847170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd\" (UID: \"4fa161cc-0cda-408d-9ecd-15d8535105c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:06:59.962108 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:06:59.962070 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:07:00.108233 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:00.108201 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd"] Jul 28 20:07:00.110715 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:07:00.110681 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa161cc_0cda_408d_9ecd_15d8535105c3.slice/crio-f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472 WatchSource:0}: Error finding container f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472: Status 404 returned error can't find the container with id f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472 Jul 28 20:07:00.517244 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:00.517210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" event={"ID":"4fa161cc-0cda-408d-9ecd-15d8535105c3","Type":"ContainerStarted","Data":"30765a906296a4d9c277285c54035c64228c136453c220cabf666af7381bdacd"} Jul 28 20:07:00.517441 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:00.517273 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" event={"ID":"4fa161cc-0cda-408d-9ecd-15d8535105c3","Type":"ContainerStarted","Data":"f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472"} Jul 28 20:07:00.536898 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:00.536846 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" podStartSLOduration=1.53683138 podStartE2EDuration="1.53683138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:00.53539997 +0000 UTC m=+1839.971353923" watchObservedRunningTime="2026-07-28 20:07:00.53683138 +0000 UTC m=+1839.972785332" Jul 28 20:07:01.524844 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:01.524811 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fa161cc-0cda-408d-9ecd-15d8535105c3" containerID="30765a906296a4d9c277285c54035c64228c136453c220cabf666af7381bdacd" exitCode=0 Jul 28 20:07:01.525249 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:01.524893 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" event={"ID":"4fa161cc-0cda-408d-9ecd-15d8535105c3","Type":"ContainerDied","Data":"30765a906296a4d9c277285c54035c64228c136453c220cabf666af7381bdacd"} Jul 28 20:07:02.672035 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:02.672009 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:07:02.776551 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:02.776516 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca\") pod \"4fa161cc-0cda-408d-9ecd-15d8535105c3\" (UID: \"4fa161cc-0cda-408d-9ecd-15d8535105c3\") " Jul 28 20:07:02.776924 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:02.776899 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fa161cc-0cda-408d-9ecd-15d8535105c3" (UID: "4fa161cc-0cda-408d-9ecd-15d8535105c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:02.877618 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:02.877527 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa161cc-0cda-408d-9ecd-15d8535105c3-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:07:03.538926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:03.538842 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" Jul 28 20:07:03.538926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:03.538866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd" event={"ID":"4fa161cc-0cda-408d-9ecd-15d8535105c3","Type":"ContainerDied","Data":"f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472"} Jul 28 20:07:03.538926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:03.538915 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8cb1a3653d9a32ec129bc594b0c12513674b427b02c95e3d1a886bb1a5c4472" Jul 28 20:07:04.972122 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:04.972087 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd"] Jul 28 20:07:04.976503 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:04.976475 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3e83fa-mphjd"] Jul 28 20:07:05.163050 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:05.163020 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa161cc-0cda-408d-9ecd-15d8535105c3" path="/var/lib/kubelet/pods/4fa161cc-0cda-408d-9ecd-15d8535105c3/volumes" Jul 28 20:07:33.940394 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.940359 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw"] Jul 28 20:07:33.941025 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.941002 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa161cc-0cda-408d-9ecd-15d8535105c3" containerName="revoke-keys" Jul 28 20:07:33.941102 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.941028 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa161cc-0cda-408d-9ecd-15d8535105c3" containerName="revoke-keys" Jul 28 20:07:33.941173 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.941159 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa161cc-0cda-408d-9ecd-15d8535105c3" containerName="revoke-keys" Jul 28 20:07:33.941233 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.941179 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c345ee-ad91-49f1-b63e-85c493571e22" containerName="istio-proxy" Jul 28 20:07:33.944363 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.944340 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:33.947121 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.947100 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:07:33.950705 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:33.950684 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw"] Jul 28 20:07:34.060017 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.059984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19b71b42-99vjw\" (UID: \"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:34.160587 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.160555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19b71b42-99vjw\" (UID: \"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:34.161248 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.161223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19b71b42-99vjw\" (UID: \"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:34.257397 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.257315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:34.631371 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.631340 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw"] Jul 28 20:07:34.632988 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:07:34.632960 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea7e5c7_4a40_4b13_8207_9ad758f9eee1.slice/crio-8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e WatchSource:0}: Error finding container 8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e: Status 404 returned error can't find the container with id 8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e Jul 28 20:07:34.718969 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:34.718939 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" event={"ID":"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1","Type":"ContainerStarted","Data":"8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e"} Jul 28 20:07:35.726184 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:35.726148 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" containerID="450589b4d7855513b29560c4b89cec7b853e0994152ace1f1b37a13f531eb4b6" exitCode=0 Jul 28 20:07:35.726611 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:35.726195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" event={"ID":"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1","Type":"ContainerDied","Data":"450589b4d7855513b29560c4b89cec7b853e0994152ace1f1b37a13f531eb4b6"} Jul 28 20:07:36.873653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:36.873627 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:36.986434 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:36.986348 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca\") pod \"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1\" (UID: \"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1\") " Jul 28 20:07:36.986830 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:36.986809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" (UID: "3ea7e5c7-4a40-4b13-8207-9ad758f9eee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:37.087633 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:37.087591 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:07:37.738238 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:37.738207 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" Jul 28 20:07:37.738416 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:37.738247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw" event={"ID":"3ea7e5c7-4a40-4b13-8207-9ad758f9eee1","Type":"ContainerDied","Data":"8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e"} Jul 28 20:07:37.738416 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:37.738295 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0db1e2cf26546830aa0ec725f0475d35fd3b9ec36249a6e9503a3ece81fb8e" Jul 28 20:07:39.915577 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:39.915529 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw"] Jul 28 20:07:39.918170 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:39.918144 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19b71b42-99vjw"] Jul 28 20:07:41.163193 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:07:41.163164 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" path="/var/lib/kubelet/pods/3ea7e5c7-4a40-4b13-8207-9ad758f9eee1/volumes" Jul 28 20:10:07.745984 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.745941 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w"] Jul 28 20:10:07.746596 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.746576 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" containerName="revoke-keys" Jul 28 20:10:07.746707 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.746599 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" containerName="revoke-keys" Jul 28 20:10:07.746772 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.746741 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea7e5c7-4a40-4b13-8207-9ad758f9eee1" containerName="revoke-keys" Jul 28 20:10:07.750022 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.749999 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:07.752563 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.752541 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:10:07.759034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.758230 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w"] Jul 28 20:10:07.782799 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.782767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9696d332-ptq8w\" (UID: \"439ab42f-879d-4c08-bcd7-16f83d7aed38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:07.883550 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.883511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9696d332-ptq8w\" (UID: \"439ab42f-879d-4c08-bcd7-16f83d7aed38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:07.884165 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:07.884146 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9696d332-ptq8w\" (UID: \"439ab42f-879d-4c08-bcd7-16f83d7aed38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:08.063221 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:08.063139 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:08.197988 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:08.197963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w"] Jul 28 20:10:08.200369 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:10:08.200329 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439ab42f_879d_4c08_bcd7_16f83d7aed38.slice/crio-eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0 WatchSource:0}: Error finding container eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0: Status 404 returned error can't find the container with id eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0 Jul 28 20:10:08.635054 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:08.635004 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" event={"ID":"439ab42f-879d-4c08-bcd7-16f83d7aed38","Type":"ContainerStarted","Data":"37e158d9233343e16a8975fe30ff6d2bac859999df0f5a569f1033a6337af7c9"} Jul 28 20:10:08.635252 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:08.635075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" event={"ID":"439ab42f-879d-4c08-bcd7-16f83d7aed38","Type":"ContainerStarted","Data":"eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0"} Jul 28 20:10:08.652050 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:08.652003 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" podStartSLOduration=1.651987481 podStartE2EDuration="1.651987481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:08.649951121 +0000 UTC m=+2028.085905071" watchObservedRunningTime="2026-07-28 20:10:08.651987481 +0000 UTC m=+2028.087941433" Jul 28 20:10:09.649132 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:09.649096 2562 generic.go:356] "Generic (PLEG): container finished" podID="439ab42f-879d-4c08-bcd7-16f83d7aed38" containerID="37e158d9233343e16a8975fe30ff6d2bac859999df0f5a569f1033a6337af7c9" exitCode=0 Jul 28 20:10:09.649538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:09.649181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" event={"ID":"439ab42f-879d-4c08-bcd7-16f83d7aed38","Type":"ContainerDied","Data":"37e158d9233343e16a8975fe30ff6d2bac859999df0f5a569f1033a6337af7c9"} Jul 28 20:10:10.794313 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:10.794287 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:10.806836 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:10.806816 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca\") pod \"439ab42f-879d-4c08-bcd7-16f83d7aed38\" (UID: \"439ab42f-879d-4c08-bcd7-16f83d7aed38\") " Jul 28 20:10:10.807112 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:10.807092 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "439ab42f-879d-4c08-bcd7-16f83d7aed38" (UID: "439ab42f-879d-4c08-bcd7-16f83d7aed38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:10:10.907529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:10.907454 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439ab42f-879d-4c08-bcd7-16f83d7aed38-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:10:11.662159 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:11.662130 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" Jul 28 20:10:11.662331 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:11.662131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w" event={"ID":"439ab42f-879d-4c08-bcd7-16f83d7aed38","Type":"ContainerDied","Data":"eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0"} Jul 28 20:10:11.662331 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:11.662255 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb153e2709aab132a9d8519825c9eeb618287c4172c7fc2075b5f5eaf2397ce0" Jul 28 20:10:18.327925 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:18.327885 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w"] Jul 28 20:10:18.331043 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:18.331018 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9696d332-ptq8w"] Jul 28 20:10:19.162547 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:10:19.162516 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439ab42f-879d-4c08-bcd7-16f83d7aed38" path="/var/lib/kubelet/pods/439ab42f-879d-4c08-bcd7-16f83d7aed38/volumes" Jul 28 20:11:21.244618 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:11:21.244588 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:11:21.254376 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:11:21.254350 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:12:08.508609 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.508571 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f"] Jul 28 20:12:08.509171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.509059 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="439ab42f-879d-4c08-bcd7-16f83d7aed38" containerName="revoke-keys" Jul 28 20:12:08.509171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.509073 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="439ab42f-879d-4c08-bcd7-16f83d7aed38" containerName="revoke-keys" Jul 28 20:12:08.509171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.509152 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="439ab42f-879d-4c08-bcd7-16f83d7aed38" containerName="revoke-keys" Jul 28 20:12:08.512281 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.512263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:08.515122 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.515096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:12:08.522267 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.522242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f"] Jul 28 20:12:08.567017 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.566989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d517388f-b6h8f\" (UID: \"5ec12d90-e9b9-44de-9c97-eb700a63fc47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:08.667560 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.667529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d517388f-b6h8f\" (UID: \"5ec12d90-e9b9-44de-9c97-eb700a63fc47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:08.668204 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.668183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d517388f-b6h8f\" (UID: \"5ec12d90-e9b9-44de-9c97-eb700a63fc47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:08.823995 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.823920 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:08.958830 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:08.958805 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f"] Jul 28 20:12:08.961653 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:12:08.961628 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec12d90_e9b9_44de_9c97_eb700a63fc47.slice/crio-edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5 WatchSource:0}: Error finding container edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5: Status 404 returned error can't find the container with id edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5 Jul 28 20:12:09.352396 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:09.352359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" event={"ID":"5ec12d90-e9b9-44de-9c97-eb700a63fc47","Type":"ContainerStarted","Data":"a11664e0e29335ce1ae94e3aaf70980bcbbe1aec3f1073c63580ee8c6d878d9e"} Jul 28 20:12:09.352573 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:09.352418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" event={"ID":"5ec12d90-e9b9-44de-9c97-eb700a63fc47","Type":"ContainerStarted","Data":"edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5"} Jul 28 20:12:09.371177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:09.371132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" podStartSLOduration=1.371116405 podStartE2EDuration="1.371116405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:12:09.369266818 +0000 UTC m=+2148.805220770" watchObservedRunningTime="2026-07-28 20:12:09.371116405 +0000 UTC m=+2148.807070356" Jul 28 20:12:10.360196 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:10.360164 2562 generic.go:356] "Generic (PLEG): container finished" podID="5ec12d90-e9b9-44de-9c97-eb700a63fc47" containerID="a11664e0e29335ce1ae94e3aaf70980bcbbe1aec3f1073c63580ee8c6d878d9e" exitCode=0 Jul 28 20:12:10.360585 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:10.360251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" event={"ID":"5ec12d90-e9b9-44de-9c97-eb700a63fc47","Type":"ContainerDied","Data":"a11664e0e29335ce1ae94e3aaf70980bcbbe1aec3f1073c63580ee8c6d878d9e"} Jul 28 20:12:11.504408 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:11.504381 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:11.591279 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:11.591245 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca\") pod \"5ec12d90-e9b9-44de-9c97-eb700a63fc47\" (UID: \"5ec12d90-e9b9-44de-9c97-eb700a63fc47\") " Jul 28 20:12:11.591600 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:11.591574 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ec12d90-e9b9-44de-9c97-eb700a63fc47" (UID: "5ec12d90-e9b9-44de-9c97-eb700a63fc47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:12:11.692008 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:11.691973 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec12d90-e9b9-44de-9c97-eb700a63fc47-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:12:12.373819 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:12.373794 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" Jul 28 20:12:12.373993 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:12.373836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f" event={"ID":"5ec12d90-e9b9-44de-9c97-eb700a63fc47","Type":"ContainerDied","Data":"edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5"} Jul 28 20:12:12.373993 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:12.373887 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd024bbbdb9e1a5e50210aede955468afd2e246131165ec3dda3314a9e010b5" Jul 28 20:12:21.370218 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:21.370149 2562 scope.go:117] "RemoveContainer" containerID="5c23b104646bff6dacfa598a7f88f8cf82180fbc476cf3019372f60567587fee" Jul 28 20:12:24.240050 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:24.240019 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f"] Jul 28 20:12:24.245131 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:24.245099 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d517388f-b6h8f"] Jul 28 20:12:25.162382 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:12:25.162351 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec12d90-e9b9-44de-9c97-eb700a63fc47" path="/var/lib/kubelet/pods/5ec12d90-e9b9-44de-9c97-eb700a63fc47/volumes" Jul 28 20:13:21.393699 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:13:21.393671 2562 scope.go:117] "RemoveContainer" containerID="30765a906296a4d9c277285c54035c64228c136453c220cabf666af7381bdacd" Jul 28 20:14:21.415988 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:21.415915 2562 scope.go:117] "RemoveContainer" containerID="450589b4d7855513b29560c4b89cec7b853e0994152ace1f1b37a13f531eb4b6" Jul 28 20:14:42.516414 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.516377 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:14:42.518123 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.518091 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec12d90-e9b9-44de-9c97-eb700a63fc47" containerName="revoke-keys" Jul 28 20:14:42.518123 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.518123 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec12d90-e9b9-44de-9c97-eb700a63fc47" containerName="revoke-keys" Jul 28 20:14:42.518442 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.518427 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec12d90-e9b9-44de-9c97-eb700a63fc47" containerName="revoke-keys" Jul 28 20:14:42.523338 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.523305 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.527103 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.527077 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ab7491-openshift-default-dockercfg-8xjz7\"" Jul 28 20:14:42.530935 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.530854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531151 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531283 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531283 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqcv\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531397 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531397 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531521 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531521 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.531629 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.531563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.532451 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.532427 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:14:42.632412 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632376 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632412 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqcv\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.632696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633035 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633035 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633035 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633035 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.632991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633210 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.633084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.633407 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.633384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.634823 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.634804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.635171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.635153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.642776 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.642748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.643454 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.643424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqcv\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv\") pod \"e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.837725 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.837641 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:42.980596 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.980568 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:14:42.981510 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:14:42.981486 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71769816_c7ec_4c91_8985_e0331e630bb0.slice/crio-0d6950ad3598ed5f5de525bd76112a2c056fec7f63e757780131e106e2dbc73e WatchSource:0}: Error finding container 0d6950ad3598ed5f5de525bd76112a2c056fec7f63e757780131e106e2dbc73e: Status 404 returned error can't find the container with id 0d6950ad3598ed5f5de525bd76112a2c056fec7f63e757780131e106e2dbc73e Jul 28 20:14:42.983801 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.983770 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:14:42.983894 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.983832 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:14:42.983894 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:42.983859 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:14:43.263965 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:43.263930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerStarted","Data":"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb"} Jul 28 20:14:43.264158 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:43.263992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerStarted","Data":"0d6950ad3598ed5f5de525bd76112a2c056fec7f63e757780131e106e2dbc73e"} Jul 28 20:14:43.283348 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:43.283292 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podStartSLOduration=1.283274875 podStartE2EDuration="1.283274875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:43.282235656 +0000 UTC m=+2302.718189620" watchObservedRunningTime="2026-07-28 20:14:43.283274875 +0000 UTC m=+2302.719228831" Jul 28 20:14:43.838570 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:43.838513 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:44.843568 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:44.843538 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:45.277400 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:45.277373 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:14:45.278300 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:14:45.278277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:15:00.155196 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.155160 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn"] Jul 28 20:15:00.158611 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.158580 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84"] Jul 28 20:15:00.158762 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.158683 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" Jul 28 20:15:00.161698 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.161675 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:15:00.162036 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.162015 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp"] Jul 28 20:15:00.162130 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.162115 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" Jul 28 20:15:00.165177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.165158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" Jul 28 20:15:00.167074 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.167053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84"] Jul 28 20:15:00.175306 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.174746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp"] Jul 28 20:15:00.176534 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.176051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" Jul 28 20:15:00.177517 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.177495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn"] Jul 28 20:15:00.184492 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.184467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" Jul 28 20:15:00.227514 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.227445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" Jul 28 20:15:00.279851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.279625 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:15:00.279851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.279745 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 28 20:15:00.368553 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.368396 2562 generic.go:356] "Generic (PLEG): container finished" podID="71769816-c7ec-4c91-8985-e0331e630bb0" containerID="208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb" exitCode=0 Jul 28 20:15:00.368553 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.368501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerDied","Data":"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb"} Jul 28 20:15:00.369060 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.369036 2562 scope.go:117] "RemoveContainer" containerID="208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb" Jul 28 20:15:00.370400 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.370362 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:15:00.370544 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.370444 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:15:00.370544 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.370478 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:15:00.579204 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.579154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn"] Jul 28 20:15:00.579898 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:15:00.579869 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9cba81_5fa8_4367_a924_677854c89356.slice/crio-d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c WatchSource:0}: Error finding container d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c: Status 404 returned error can't find the container with id d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c Jul 28 20:15:00.583492 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.583379 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84"] Jul 28 20:15:00.583570 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:15:00.583529 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd694b0b4_b126_44fe_bcc8_efdc841e2902.slice/crio-8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b WatchSource:0}: Error finding container 8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b: Status 404 returned error can't find the container with id 8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b Jul 28 20:15:00.634272 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:00.634248 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp"] Jul 28 20:15:00.635260 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:15:00.635231 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f06b4b7_a9b1_421e_9524_d125756cfe2a.slice/crio-60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07 WatchSource:0}: Error finding container 60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07: Status 404 returned error can't find the container with id 60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07 Jul 28 20:15:01.376412 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.376380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" event={"ID":"9a9cba81-5fa8-4367-a924-677854c89356","Type":"ContainerStarted","Data":"fea5dba655e56cc326337def5272f3e9a1922b7271755bb0c47bcf85c2d228c4"} Jul 28 20:15:01.376891 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.376439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" event={"ID":"9a9cba81-5fa8-4367-a924-677854c89356","Type":"ContainerStarted","Data":"d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c"} Jul 28 20:15:01.377990 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.377959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" event={"ID":"1f06b4b7-a9b1-421e-9524-d125756cfe2a","Type":"ContainerStarted","Data":"9110b6a6a0aab45ef2c07695005c4aa3ee1ad36090d5018055464bf393e427d1"} Jul 28 20:15:01.378129 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.378010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" event={"ID":"1f06b4b7-a9b1-421e-9524-d125756cfe2a","Type":"ContainerStarted","Data":"60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07"} Jul 28 20:15:01.379935 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.379912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerStarted","Data":"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e"} Jul 28 20:15:01.381350 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.381328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" event={"ID":"d694b0b4-b126-44fe-bcc8-efdc841e2902","Type":"ContainerStarted","Data":"d02ddf09cdc247b32f865e658fd0ad212bc3cb39725a999db673d1f5161c6de8"} Jul 28 20:15:01.381434 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.381371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" event={"ID":"d694b0b4-b126-44fe-bcc8-efdc841e2902","Type":"ContainerStarted","Data":"8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b"} Jul 28 20:15:01.398953 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.398910 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" podStartSLOduration=1.398896768 podStartE2EDuration="1.398896768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:01.396237736 +0000 UTC m=+2320.832191685" watchObservedRunningTime="2026-07-28 20:15:01.398896768 +0000 UTC m=+2320.834850720" Jul 28 20:15:01.415599 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.415556 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" podStartSLOduration=1.41554273 podStartE2EDuration="1.41554273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:01.414752151 +0000 UTC m=+2320.850706104" watchObservedRunningTime="2026-07-28 20:15:01.41554273 +0000 UTC m=+2320.851496682" Jul 28 20:15:01.461686 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.458354 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" podStartSLOduration=1.458329958 podStartE2EDuration="1.458329958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:01.44997431 +0000 UTC m=+2320.885928288" watchObservedRunningTime="2026-07-28 20:15:01.458329958 +0000 UTC m=+2320.894283912" Jul 28 20:15:01.838461 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.838422 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:15:01.838933 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.838911 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:15:01.839003 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:01.838971 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 28 20:15:02.388877 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.388843 2562 generic.go:356] "Generic (PLEG): container finished" podID="1f06b4b7-a9b1-421e-9524-d125756cfe2a" containerID="9110b6a6a0aab45ef2c07695005c4aa3ee1ad36090d5018055464bf393e427d1" exitCode=0 Jul 28 20:15:02.389365 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.388924 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" event={"ID":"1f06b4b7-a9b1-421e-9524-d125756cfe2a","Type":"ContainerDied","Data":"9110b6a6a0aab45ef2c07695005c4aa3ee1ad36090d5018055464bf393e427d1"} Jul 28 20:15:02.390512 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.390478 2562 generic.go:356] "Generic (PLEG): container finished" podID="d694b0b4-b126-44fe-bcc8-efdc841e2902" containerID="d02ddf09cdc247b32f865e658fd0ad212bc3cb39725a999db673d1f5161c6de8" exitCode=0 Jul 28 20:15:02.390633 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.390571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" event={"ID":"d694b0b4-b126-44fe-bcc8-efdc841e2902","Type":"ContainerDied","Data":"d02ddf09cdc247b32f865e658fd0ad212bc3cb39725a999db673d1f5161c6de8"} Jul 28 20:15:02.392087 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.392068 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a9cba81-5fa8-4367-a924-677854c89356" containerID="fea5dba655e56cc326337def5272f3e9a1922b7271755bb0c47bcf85c2d228c4" exitCode=0 Jul 28 20:15:02.392177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.392127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" event={"ID":"9a9cba81-5fa8-4367-a924-677854c89356","Type":"ContainerDied","Data":"fea5dba655e56cc326337def5272f3e9a1922b7271755bb0c47bcf85c2d228c4"} Jul 28 20:15:02.838644 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.838604 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:15:02.838859 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:02.838676 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 28 20:15:03.588051 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:03.588026 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" Jul 28 20:15:03.674547 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:03.674416 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" Jul 28 20:15:03.678350 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:03.678333 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" Jul 28 20:15:03.838201 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:03.838167 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:15:03.838377 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:03.838246 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 28 20:15:04.405282 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.405252 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" Jul 28 20:15:04.405493 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.405248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84" event={"ID":"d694b0b4-b126-44fe-bcc8-efdc841e2902","Type":"ContainerDied","Data":"8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b"} Jul 28 20:15:04.405493 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.405417 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff07a5f01853602c870b8e906c8e760bddb868b24d301af2f661605238b0a9b" Jul 28 20:15:04.406744 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.406722 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" Jul 28 20:15:04.406864 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.406764 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4tcgn" event={"ID":"9a9cba81-5fa8-4367-a924-677854c89356","Type":"ContainerDied","Data":"d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c"} Jul 28 20:15:04.406864 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.406805 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ad55e5da7a03a7dd765659a67d326d3074044d6581011ecffce9e98964034c" Jul 28 20:15:04.408171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.408152 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" event={"ID":"1f06b4b7-a9b1-421e-9524-d125756cfe2a","Type":"ContainerDied","Data":"60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07"} Jul 28 20:15:04.408273 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.408184 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b00c0dc0de60352fe38762a944c02c497134abb76ce894d0744d5e533d6d07" Jul 28 20:15:04.408273 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.408185 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp" Jul 28 20:15:04.838504 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.838414 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:15:04.838504 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:04.838467 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 28 20:15:05.842550 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:05.842518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:15:05.842962 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:05.842879 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:15:05.843565 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:15:05.843547 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:16:21.284884 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:21.284857 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:16:21.302585 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:21.302555 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:16:21.447786 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:21.447756 2562 scope.go:117] "RemoveContainer" containerID="37e158d9233343e16a8975fe30ff6d2bac859999df0f5a569f1033a6337af7c9" Jul 28 20:16:25.921538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.921500 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb"] Jul 28 20:16:25.922157 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922134 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f06b4b7-a9b1-421e-9524-d125756cfe2a" containerName="cleanup" Jul 28 20:16:25.922246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922159 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f06b4b7-a9b1-421e-9524-d125756cfe2a" containerName="cleanup" Jul 28 20:16:25.922246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922209 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9cba81-5fa8-4367-a924-677854c89356" containerName="cleanup" Jul 28 20:16:25.922246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922218 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9cba81-5fa8-4367-a924-677854c89356" containerName="cleanup" Jul 28 20:16:25.922246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922227 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d694b0b4-b126-44fe-bcc8-efdc841e2902" containerName="cleanup" Jul 28 20:16:25.922246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922236 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d694b0b4-b126-44fe-bcc8-efdc841e2902" containerName="cleanup" Jul 28 20:16:25.922508 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922367 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9cba81-5fa8-4367-a924-677854c89356" containerName="cleanup" Jul 28 20:16:25.922508 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922382 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f06b4b7-a9b1-421e-9524-d125756cfe2a" containerName="cleanup" Jul 28 20:16:25.922508 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.922396 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d694b0b4-b126-44fe-bcc8-efdc841e2902" containerName="cleanup" Jul 28 20:16:25.927065 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.927043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:25.929878 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.929856 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:16:25.933813 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:25.933790 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb"] Jul 28 20:16:26.028327 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.028291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb\" (UID: \"d622ea25-9ae8-43f7-b389-e8d57aee435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:26.129783 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.129751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb\" (UID: \"d622ea25-9ae8-43f7-b389-e8d57aee435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:26.130354 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.130335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb\" (UID: \"d622ea25-9ae8-43f7-b389-e8d57aee435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:26.239575 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.239472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:26.385817 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.385783 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb"] Jul 28 20:16:26.388048 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:16:26.388010 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd622ea25_9ae8_43f7_b389_e8d57aee435c.slice/crio-b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131 WatchSource:0}: Error finding container b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131: Status 404 returned error can't find the container with id b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131 Jul 28 20:16:26.906187 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.906148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" event={"ID":"d622ea25-9ae8-43f7-b389-e8d57aee435c","Type":"ContainerStarted","Data":"250c3c91167b692224e630f99f578657bb740df8ba6eafc2a0559b8456955b3d"} Jul 28 20:16:26.906378 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.906213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" event={"ID":"d622ea25-9ae8-43f7-b389-e8d57aee435c","Type":"ContainerStarted","Data":"b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131"} Jul 28 20:16:26.922283 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:26.922238 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" podStartSLOduration=1.9222227090000001 podStartE2EDuration="1.922222709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:16:26.920327681 +0000 UTC m=+2406.356281653" watchObservedRunningTime="2026-07-28 20:16:26.922222709 +0000 UTC m=+2406.358176683" Jul 28 20:16:27.914343 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:27.914308 2562 generic.go:356] "Generic (PLEG): container finished" podID="d622ea25-9ae8-43f7-b389-e8d57aee435c" containerID="250c3c91167b692224e630f99f578657bb740df8ba6eafc2a0559b8456955b3d" exitCode=0 Jul 28 20:16:27.914555 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:27.914397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" event={"ID":"d622ea25-9ae8-43f7-b389-e8d57aee435c","Type":"ContainerDied","Data":"250c3c91167b692224e630f99f578657bb740df8ba6eafc2a0559b8456955b3d"} Jul 28 20:16:29.069839 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.069812 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:29.162199 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.162163 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca\") pod \"d622ea25-9ae8-43f7-b389-e8d57aee435c\" (UID: \"d622ea25-9ae8-43f7-b389-e8d57aee435c\") " Jul 28 20:16:29.162560 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.162535 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d622ea25-9ae8-43f7-b389-e8d57aee435c" (UID: "d622ea25-9ae8-43f7-b389-e8d57aee435c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:16:29.263177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.263139 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d622ea25-9ae8-43f7-b389-e8d57aee435c-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:16:29.928995 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.928967 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" Jul 28 20:16:29.928995 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.928977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb" event={"ID":"d622ea25-9ae8-43f7-b389-e8d57aee435c","Type":"ContainerDied","Data":"b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131"} Jul 28 20:16:29.929222 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:29.929025 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72b272a5e8b540d1f5114e3d427e9553ac4e420cd4c9b2455449aa63eadf131" Jul 28 20:16:36.430334 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:36.430301 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp"] Jul 28 20:16:36.433249 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:36.433223 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-24876e-29754495-vmsgp"] Jul 28 20:16:36.625000 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:36.624965 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb"] Jul 28 20:16:36.627515 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:36.627492 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-24876e-mtrsb"] Jul 28 20:16:37.162977 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:37.162938 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f06b4b7-a9b1-421e-9524-d125756cfe2a" path="/var/lib/kubelet/pods/1f06b4b7-a9b1-421e-9524-d125756cfe2a/volumes" Jul 28 20:16:37.163357 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:37.163337 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d622ea25-9ae8-43f7-b389-e8d57aee435c" path="/var/lib/kubelet/pods/d622ea25-9ae8-43f7-b389-e8d57aee435c/volumes" Jul 28 20:16:43.500163 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.500080 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl"] Jul 28 20:16:43.500529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.500501 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d622ea25-9ae8-43f7-b389-e8d57aee435c" containerName="revoke-keys" Jul 28 20:16:43.500529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.500511 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d622ea25-9ae8-43f7-b389-e8d57aee435c" containerName="revoke-keys" Jul 28 20:16:43.500602 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.500569 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d622ea25-9ae8-43f7-b389-e8d57aee435c" containerName="revoke-keys" Jul 28 20:16:43.504798 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.504781 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:43.507585 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.507559 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:16:43.510973 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.510948 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl"] Jul 28 20:16:43.598216 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.598184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl\" (UID: \"b9d2d403-aaad-4fb8-ad2e-bef80a154eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:43.698882 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.698841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl\" (UID: \"b9d2d403-aaad-4fb8-ad2e-bef80a154eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:43.699501 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.699474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl\" (UID: \"b9d2d403-aaad-4fb8-ad2e-bef80a154eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:43.819438 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.819349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:43.962464 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:43.962433 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl"] Jul 28 20:16:43.964202 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:16:43.964177 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d2d403_aaad_4fb8_ad2e_bef80a154eee.slice/crio-8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460 WatchSource:0}: Error finding container 8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460: Status 404 returned error can't find the container with id 8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460 Jul 28 20:16:44.015721 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:44.015692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" event={"ID":"b9d2d403-aaad-4fb8-ad2e-bef80a154eee","Type":"ContainerStarted","Data":"8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460"} Jul 28 20:16:45.023165 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:45.023080 2562 generic.go:356] "Generic (PLEG): container finished" podID="b9d2d403-aaad-4fb8-ad2e-bef80a154eee" containerID="2715770bc2bfd420cfbb86e553bb6139b041a31d84e18d3baaa58804e3d4bea6" exitCode=0 Jul 28 20:16:45.023530 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:45.023165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" event={"ID":"b9d2d403-aaad-4fb8-ad2e-bef80a154eee","Type":"ContainerDied","Data":"2715770bc2bfd420cfbb86e553bb6139b041a31d84e18d3baaa58804e3d4bea6"} Jul 28 20:16:46.161202 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:46.161176 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:46.224241 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:46.224206 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca\") pod \"b9d2d403-aaad-4fb8-ad2e-bef80a154eee\" (UID: \"b9d2d403-aaad-4fb8-ad2e-bef80a154eee\") " Jul 28 20:16:46.224707 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:46.224614 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9d2d403-aaad-4fb8-ad2e-bef80a154eee" (UID: "b9d2d403-aaad-4fb8-ad2e-bef80a154eee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:16:46.325644 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:46.325547 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2d403-aaad-4fb8-ad2e-bef80a154eee-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:16:47.035147 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:47.035115 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" Jul 28 20:16:47.035147 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:47.035126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl" event={"ID":"b9d2d403-aaad-4fb8-ad2e-bef80a154eee","Type":"ContainerDied","Data":"8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460"} Jul 28 20:16:47.035361 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:47.035169 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce046c8abd9bd00fbfe558ec81d562bc392819881567490e55854365084b460" Jul 28 20:16:54.419679 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:54.419632 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84"] Jul 28 20:16:54.431566 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:54.431535 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7491-29754495-vlx84"] Jul 28 20:16:54.693134 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:54.693034 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl"] Jul 28 20:16:54.696326 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:54.696294 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7491-6zgpl"] Jul 28 20:16:55.163958 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:55.163925 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d2d403-aaad-4fb8-ad2e-bef80a154eee" path="/var/lib/kubelet/pods/b9d2d403-aaad-4fb8-ad2e-bef80a154eee/volumes" Jul 28 20:16:55.164323 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:16:55.164303 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d694b0b4-b126-44fe-bcc8-efdc841e2902" path="/var/lib/kubelet/pods/d694b0b4-b126-44fe-bcc8-efdc841e2902/volumes" Jul 28 20:17:01.174510 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:01.174480 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:17:01.174995 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:01.174683 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" containerID="cri-o://97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e" gracePeriod=30 Jul 28 20:17:05.845947 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:05.845900 2562 patch_prober.go:28] interesting pod/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 20:17:05.846458 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:05.845980 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 20:17:06.443818 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.443761 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:17:06.621337 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621369 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621394 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621422 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621501 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qqcv\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621592 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket\") pod \"71769816-c7ec-4c91-8985-e0331e630bb0\" (UID: \"71769816-c7ec-4c91-8985-e0331e630bb0\") " Jul 28 20:17:06.621921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621790 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:17:06.621921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621895 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71769816-c7ec-4c91-8985-e0331e630bb0-istiod-ca-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.622143 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.621984 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:17:06.622143 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.622017 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:17:06.622263 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.622172 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data" (OuterVolumeSpecName: "istio-data") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:17:06.622319 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.622255 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:17:06.623760 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.623731 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:17:06.623870 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.623761 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token" (OuterVolumeSpecName: "istio-token") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:17:06.623870 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.623840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:17:06.624012 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.623994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv" (OuterVolumeSpecName: "kube-api-access-9qqcv") pod "71769816-c7ec-4c91-8985-e0331e630bb0" (UID: "71769816-c7ec-4c91-8985-e0331e630bb0"). InnerVolumeSpecName "kube-api-access-9qqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:17:06.723366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723287 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-data\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723322 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-istio-envoy\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723337 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723350 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723366 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723364 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-workload-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723377 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71769816-c7ec-4c91-8985-e0331e630bb0-istio-podinfo\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723389 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqcv\" (UniqueName: \"kubernetes.io/projected/71769816-c7ec-4c91-8985-e0331e630bb0-kube-api-access-9qqcv\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:06.723634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:06.723403 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71769816-c7ec-4c91-8985-e0331e630bb0-credential-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:17:07.165975 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.165946 2562 generic.go:356] "Generic (PLEG): container finished" podID="71769816-c7ec-4c91-8985-e0331e630bb0" containerID="97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e" exitCode=0 Jul 28 20:17:07.166436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.165986 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" Jul 28 20:17:07.166436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.166029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerDied","Data":"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e"} Jul 28 20:17:07.166436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.166079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7" event={"ID":"71769816-c7ec-4c91-8985-e0331e630bb0","Type":"ContainerDied","Data":"0d6950ad3598ed5f5de525bd76112a2c056fec7f63e757780131e106e2dbc73e"} Jul 28 20:17:07.166436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.166140 2562 scope.go:117] "RemoveContainer" containerID="97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e" Jul 28 20:17:07.179998 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.179973 2562 scope.go:117] "RemoveContainer" containerID="208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb" Jul 28 20:17:07.188753 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.188734 2562 scope.go:117] "RemoveContainer" containerID="97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e" Jul 28 20:17:07.189012 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:17:07.188987 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e\": container with ID starting with 97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e not found: ID does not exist" containerID="97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e" Jul 28 20:17:07.189064 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.189021 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e"} err="failed to get container status \"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e\": rpc error: code = NotFound desc = could not find container \"97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e\": container with ID starting with 97756cbcb4c18fa663f540c9e0105761ec71fd68bd8619212a3b7d5f34be143e not found: ID does not exist" Jul 28 20:17:07.189064 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.189043 2562 scope.go:117] "RemoveContainer" containerID="208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb" Jul 28 20:17:07.189377 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:17:07.189338 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb\": container with ID starting with 208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb not found: ID does not exist" containerID="208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb" Jul 28 20:17:07.189486 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.189373 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb"} err="failed to get container status \"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb\": rpc error: code = NotFound desc = could not find container \"208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb\": container with ID starting with 208ec088eca06170e0a689ed5220b24913cad97aec6a2cc2bc96bc45cb5e46eb not found: ID does not exist" Jul 28 20:17:07.192517 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.192499 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:17:07.196403 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:07.196384 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ab7491-openshift-default-7647d68f6d-fgjb7"] Jul 28 20:17:09.163403 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:09.163359 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" path="/var/lib/kubelet/pods/71769816-c7ec-4c91-8985-e0331e630bb0/volumes" Jul 28 20:17:30.501226 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501196 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501611 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501623 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501630 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d2d403-aaad-4fb8-ad2e-bef80a154eee" containerName="revoke-keys" Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501635 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d2d403-aaad-4fb8-ad2e-bef80a154eee" containerName="revoke-keys" Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501651 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501670 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501907 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501732 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501907 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501741 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71769816-c7ec-4c91-8985-e0331e630bb0" containerName="istio-proxy" Jul 28 20:17:30.501907 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.501748 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d2d403-aaad-4fb8-ad2e-bef80a154eee" containerName="revoke-keys" Jul 28 20:17:30.504843 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.504827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.507953 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.507932 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bf09e7-openshift-default-dockercfg-6bmdl\"" Jul 28 20:17:30.543045 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.543021 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:17:30.636715 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.636883 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.636883 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.636883 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.636883 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.637092 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.637092 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.636931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkst\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.637092 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.637000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.637092 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.637031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.737815 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkst\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.737952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738562 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738562 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.738651 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.738621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.740482 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.740455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.740815 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.740794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.746517 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.746497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.746719 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.746699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkst\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst\") pod \"e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.815501 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.815433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:30.965046 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.965021 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:17:30.967294 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:17:30.967266 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992f3da5_6b0e_466b_bb17_8b9a67b743aa.slice/crio-791f19456b44ff0f3b007f4647e38c89414d158a5b733f6336d0b02d662b5362 WatchSource:0}: Error finding container 791f19456b44ff0f3b007f4647e38c89414d158a5b733f6336d0b02d662b5362: Status 404 returned error can't find the container with id 791f19456b44ff0f3b007f4647e38c89414d158a5b733f6336d0b02d662b5362 Jul 28 20:17:30.969447 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.969415 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:30.969545 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.969497 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:30.969609 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:30.969544 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:31.308971 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:31.308932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerStarted","Data":"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853"} Jul 28 20:17:31.309164 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:31.308993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerStarted","Data":"791f19456b44ff0f3b007f4647e38c89414d158a5b733f6336d0b02d662b5362"} Jul 28 20:17:31.335147 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:31.335099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podStartSLOduration=1.335083904 podStartE2EDuration="1.335083904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:31.331924307 +0000 UTC m=+2470.767878258" watchObservedRunningTime="2026-07-28 20:17:31.335083904 +0000 UTC m=+2470.771037856" Jul 28 20:17:31.816387 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:31.816344 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:32.821209 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:32.821179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:33.321383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:33.321353 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:33.322264 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:33.322244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:44.405254 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.405224 2562 generic.go:356] "Generic (PLEG): container finished" podID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerID="6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853" exitCode=0 Jul 28 20:17:44.405682 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.405298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerDied","Data":"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853"} Jul 28 20:17:44.405759 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.405741 2562 scope.go:117] "RemoveContainer" containerID="6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853" Jul 28 20:17:44.406752 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.406725 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:44.406833 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.406790 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:44.406833 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:44.406828 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:17:45.413582 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:45.413544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerStarted","Data":"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b"} Jul 28 20:17:45.815805 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:45.815720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:45.817092 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:45.817066 2562 patch_prober.go:28] interesting pod/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Jul 28 20:17:45.817245 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:45.817121 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Jul 28 20:17:46.816554 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:46.816520 2562 patch_prober.go:28] interesting pod/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Jul 28 20:17:46.817011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:46.816579 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Jul 28 20:17:47.815823 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:47.815785 2562 patch_prober.go:28] interesting pod/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Jul 28 20:17:47.816010 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:47.815843 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Jul 28 20:17:48.321872 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:48.321838 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:48.816046 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:48.816009 2562 patch_prober.go:28] interesting pod/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Jul 28 20:17:48.816234 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:48.816060 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Jul 28 20:17:49.819700 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:49.819652 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:17:49.820840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:17:49.820816 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:18:01.241270 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.241233 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr"] Jul 28 20:18:01.244178 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.244151 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:01.246646 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.246623 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:18:01.253012 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.252986 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr"] Jul 28 20:18:01.319829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.319803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr\" (UID: \"a5be6563-770f-418e-a30b-829ce1678f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:01.420632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.420601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr\" (UID: \"a5be6563-770f-418e-a30b-829ce1678f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:01.421187 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.421166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr\" (UID: \"a5be6563-770f-418e-a30b-829ce1678f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:01.558586 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.558499 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:01.700998 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:01.700969 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr"] Jul 28 20:18:01.703002 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:18:01.702971 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5be6563_770f_418e_a30b_829ce1678f31.slice/crio-a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0 WatchSource:0}: Error finding container a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0: Status 404 returned error can't find the container with id a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0 Jul 28 20:18:02.527886 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:02.527853 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5be6563-770f-418e-a30b-829ce1678f31" containerID="825474a14ff982b6ac4d2fdfe4bfb1d29b879d216a76ecc2b7bb4777f9e95450" exitCode=0 Jul 28 20:18:02.528235 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:02.527942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" event={"ID":"a5be6563-770f-418e-a30b-829ce1678f31","Type":"ContainerDied","Data":"825474a14ff982b6ac4d2fdfe4bfb1d29b879d216a76ecc2b7bb4777f9e95450"} Jul 28 20:18:02.528235 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:02.528033 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" event={"ID":"a5be6563-770f-418e-a30b-829ce1678f31","Type":"ContainerStarted","Data":"a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0"} Jul 28 20:18:03.673644 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:03.673620 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:03.740357 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:03.740312 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca\") pod \"a5be6563-770f-418e-a30b-829ce1678f31\" (UID: \"a5be6563-770f-418e-a30b-829ce1678f31\") " Jul 28 20:18:03.740630 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:03.740610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5be6563-770f-418e-a30b-829ce1678f31" (UID: "a5be6563-770f-418e-a30b-829ce1678f31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:18:03.740737 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:03.740724 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5be6563-770f-418e-a30b-829ce1678f31-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:04.541193 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:04.541164 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" Jul 28 20:18:04.541367 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:04.541202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr" event={"ID":"a5be6563-770f-418e-a30b-829ce1678f31","Type":"ContainerDied","Data":"a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0"} Jul 28 20:18:04.541367 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:04.541249 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e4d56ad51137e9bc50afbba80c446fd97653048cdfd69cc1faa487e37d15e0" Jul 28 20:18:16.851647 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:16.851522 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr"] Jul 28 20:18:16.857653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:16.857619 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a5ccdf-ckmbr"] Jul 28 20:18:17.162940 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:17.162904 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5be6563-770f-418e-a30b-829ce1678f31" path="/var/lib/kubelet/pods/a5be6563-770f-418e-a30b-829ce1678f31/volumes" Jul 28 20:18:21.485840 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:21.485812 2562 scope.go:117] "RemoveContainer" containerID="a11664e0e29335ce1ae94e3aaf70980bcbbe1aec3f1073c63580ee8c6d878d9e" Jul 28 20:18:23.686403 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.686360 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq"] Jul 28 20:18:23.686922 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.686847 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5be6563-770f-418e-a30b-829ce1678f31" containerName="revoke-keys" Jul 28 20:18:23.686922 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.686861 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5be6563-770f-418e-a30b-829ce1678f31" containerName="revoke-keys" Jul 28 20:18:23.687041 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.686937 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5be6563-770f-418e-a30b-829ce1678f31" containerName="revoke-keys" Jul 28 20:18:23.690185 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.690162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:23.692455 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.692435 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:18:23.696436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.696362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq"] Jul 28 20:18:23.812981 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.812951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq\" (UID: \"68eacb72-6706-4d71-acaa-b00839cb5c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:23.913866 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.913833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq\" (UID: \"68eacb72-6706-4d71-acaa-b00839cb5c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:23.914446 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:23.914426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq\" (UID: \"68eacb72-6706-4d71-acaa-b00839cb5c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:24.001540 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:24.001471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:24.174389 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:24.174370 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq"] Jul 28 20:18:24.176208 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:18:24.176184 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68eacb72_6706_4d71_acaa_b00839cb5c70.slice/crio-a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6 WatchSource:0}: Error finding container a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6: Status 404 returned error can't find the container with id a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6 Jul 28 20:18:24.671749 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:24.671712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" event={"ID":"68eacb72-6706-4d71-acaa-b00839cb5c70","Type":"ContainerStarted","Data":"f5c38074153782e925b32c08de8b4791315ad273d93db286c070262006724688"} Jul 28 20:18:24.671903 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:24.671774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" event={"ID":"68eacb72-6706-4d71-acaa-b00839cb5c70","Type":"ContainerStarted","Data":"a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6"} Jul 28 20:18:24.687346 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:24.687302 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" podStartSLOduration=1.6872871649999999 podStartE2EDuration="1.687287165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:24.686102838 +0000 UTC m=+2524.122056791" watchObservedRunningTime="2026-07-28 20:18:24.687287165 +0000 UTC m=+2524.123241116" Jul 28 20:18:25.679171 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:25.679140 2562 generic.go:356] "Generic (PLEG): container finished" podID="68eacb72-6706-4d71-acaa-b00839cb5c70" containerID="f5c38074153782e925b32c08de8b4791315ad273d93db286c070262006724688" exitCode=0 Jul 28 20:18:25.679346 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:25.679232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" event={"ID":"68eacb72-6706-4d71-acaa-b00839cb5c70","Type":"ContainerDied","Data":"f5c38074153782e925b32c08de8b4791315ad273d93db286c070262006724688"} Jul 28 20:18:26.822962 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:26.822936 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:26.945920 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:26.945841 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca\") pod \"68eacb72-6706-4d71-acaa-b00839cb5c70\" (UID: \"68eacb72-6706-4d71-acaa-b00839cb5c70\") " Jul 28 20:18:26.946265 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:26.946241 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68eacb72-6706-4d71-acaa-b00839cb5c70" (UID: "68eacb72-6706-4d71-acaa-b00839cb5c70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:18:27.047124 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:27.047091 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68eacb72-6706-4d71-acaa-b00839cb5c70-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:27.692341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:27.692312 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" Jul 28 20:18:27.692540 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:27.692356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq" event={"ID":"68eacb72-6706-4d71-acaa-b00839cb5c70","Type":"ContainerDied","Data":"a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6"} Jul 28 20:18:27.692540 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:27.692403 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a363cad2e28e7a262fb665281af3a0a546e3cbb35bf50097064175f21a99add6" Jul 28 20:18:39.273062 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:39.273025 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq"] Jul 28 20:18:39.275681 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:39.275643 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf09e7-bl7qq"] Jul 28 20:18:41.163039 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:41.163004 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68eacb72-6706-4d71-acaa-b00839cb5c70" path="/var/lib/kubelet/pods/68eacb72-6706-4d71-acaa-b00839cb5c70/volumes" Jul 28 20:18:45.869424 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:45.869390 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:18:45.869923 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:45.869631 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" containerID="cri-o://97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b" gracePeriod=30 Jul 28 20:18:49.822714 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:49.822653 2562 patch_prober.go:28] interesting pod/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 20:18:49.823096 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:49.822730 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 20:18:51.151627 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.151600 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:18:51.168169 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168146 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168331 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168331 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168235 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168331 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168309 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168375 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168406 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168419 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168466 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168494 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgkst\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst\") pod \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\" (UID: \"992f3da5-6b0e-466b-bb17-8b9a67b743aa\") " Jul 28 20:18:51.168834 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168603 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:18:51.168834 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168752 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:18:51.168960 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168846 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-credential-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.168960 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168864 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.168960 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.168878 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-workload-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.169111 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.169070 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:18:51.169251 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.169231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:18:51.170897 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.170865 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:18:51.171086 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.171059 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst" (OuterVolumeSpecName: "kube-api-access-lgkst") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "kube-api-access-lgkst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:18:51.171086 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.171076 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:18:51.171212 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.171084 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "992f3da5-6b0e-466b-bb17-8b9a67b743aa" (UID: "992f3da5-6b0e-466b-bb17-8b9a67b743aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:18:51.269702 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269653 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.269702 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269697 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-data\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.269702 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269707 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-envoy\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.269934 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269716 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istio-podinfo\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.269934 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269724 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992f3da5-6b0e-466b-bb17-8b9a67b743aa-istiod-ca-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.269934 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.269734 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkst\" (UniqueName: \"kubernetes.io/projected/992f3da5-6b0e-466b-bb17-8b9a67b743aa-kube-api-access-lgkst\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:18:51.848270 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.848240 2562 generic.go:356] "Generic (PLEG): container finished" podID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerID="97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b" exitCode=0 Jul 28 20:18:51.848494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.848294 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" Jul 28 20:18:51.848494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.848326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerDied","Data":"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b"} Jul 28 20:18:51.848494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.848374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l" event={"ID":"992f3da5-6b0e-466b-bb17-8b9a67b743aa","Type":"ContainerDied","Data":"791f19456b44ff0f3b007f4647e38c89414d158a5b733f6336d0b02d662b5362"} Jul 28 20:18:51.848494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.848395 2562 scope.go:117] "RemoveContainer" containerID="97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b" Jul 28 20:18:51.859649 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.859631 2562 scope.go:117] "RemoveContainer" containerID="6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853" Jul 28 20:18:51.868030 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.868015 2562 scope.go:117] "RemoveContainer" containerID="97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b" Jul 28 20:18:51.868268 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:18:51.868250 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b\": container with ID starting with 97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b not found: ID does not exist" containerID="97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b" Jul 28 20:18:51.868317 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.868279 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b"} err="failed to get container status \"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b\": rpc error: code = NotFound desc = could not find container \"97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b\": container with ID starting with 97157e05e6f81f86f833d8182b2fce1b318810f9cf5870010faa57c75b7b2e2b not found: ID does not exist" Jul 28 20:18:51.868317 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.868295 2562 scope.go:117] "RemoveContainer" containerID="6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853" Jul 28 20:18:51.868517 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:18:51.868503 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853\": container with ID starting with 6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853 not found: ID does not exist" containerID="6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853" Jul 28 20:18:51.868569 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.868521 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853"} err="failed to get container status \"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853\": rpc error: code = NotFound desc = could not find container \"6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853\": container with ID starting with 6e00e4c6edf009a1b8b01336a0721aa97d61563914bb43a70299e876c6086853 not found: ID does not exist" Jul 28 20:18:51.873429 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.873405 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:18:51.881052 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:51.881026 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bf09e7-openshift-default-7cc7b6cd64-ww25l"] Jul 28 20:18:53.163809 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:18:53.163778 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" path="/var/lib/kubelet/pods/992f3da5-6b0e-466b-bb17-8b9a67b743aa/volumes" Jul 28 20:19:15.910743 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.910711 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911138 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911152 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911175 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911181 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911195 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68eacb72-6706-4d71-acaa-b00839cb5c70" containerName="revoke-keys" Jul 28 20:19:15.911256 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911200 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="68eacb72-6706-4d71-acaa-b00839cb5c70" containerName="revoke-keys" Jul 28 20:19:15.911459 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911263 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.911459 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911271 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="68eacb72-6706-4d71-acaa-b00839cb5c70" containerName="revoke-keys" Jul 28 20:19:15.911459 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.911276 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="992f3da5-6b0e-466b-bb17-8b9a67b743aa" containerName="istio-proxy" Jul 28 20:19:15.914592 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.914573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:15.917221 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.917203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-47f87c-openshift-default-dockercfg-bgqtf\"" Jul 28 20:19:15.927418 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:15.927395 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:19:16.007440 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007440 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8tj\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007655 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007629 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.007882 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.007710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108629 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.108829 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109095 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109095 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8tj\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109095 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.108928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109250 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.109129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109250 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.109223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109362 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.109273 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109362 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.109294 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.109498 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.109479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.111001 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.110982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.111284 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.111268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.117070 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.117046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.117259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.117238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8tj\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj\") pod \"e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.225993 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.225909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:16.372915 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.372888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:19:16.374817 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:19:16.374778 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17b36cf_8bc4_4b2f_aa27_6b0cf337e72d.slice/crio-64486442395cfed1209a5c3b6482de6555112dcfdd56bf2cb16683d736594e3d WatchSource:0}: Error finding container 64486442395cfed1209a5c3b6482de6555112dcfdd56bf2cb16683d736594e3d: Status 404 returned error can't find the container with id 64486442395cfed1209a5c3b6482de6555112dcfdd56bf2cb16683d736594e3d Jul 28 20:19:16.377145 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.377113 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:16.377215 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.377195 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:16.377262 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:16.377244 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:17.000708 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:17.000644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerStarted","Data":"e708eafc8c1f7f93de0a40c29844851032536139bd59be50e79b389ef733b828"} Jul 28 20:19:17.001153 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:17.000732 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerStarted","Data":"64486442395cfed1209a5c3b6482de6555112dcfdd56bf2cb16683d736594e3d"} Jul 28 20:19:17.025960 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:17.025916 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podStartSLOduration=2.025898624 podStartE2EDuration="2.025898624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:17.023960095 +0000 UTC m=+2576.459914049" watchObservedRunningTime="2026-07-28 20:19:17.025898624 +0000 UTC m=+2576.461852576" Jul 28 20:19:17.226447 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:17.226404 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:17.231542 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:17.231518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:18.007404 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:18.007372 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:18.008530 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:18.008510 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:34.049870 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:34.049821 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 20:19:34.050420 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:34.049914 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 20:19:35.119700 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.119610 2562 generic.go:356] "Generic (PLEG): container finished" podID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerID="e708eafc8c1f7f93de0a40c29844851032536139bd59be50e79b389ef733b828" exitCode=0 Jul 28 20:19:35.120065 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.119717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerDied","Data":"e708eafc8c1f7f93de0a40c29844851032536139bd59be50e79b389ef733b828"} Jul 28 20:19:35.120132 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.120114 2562 scope.go:117] "RemoveContainer" containerID="e708eafc8c1f7f93de0a40c29844851032536139bd59be50e79b389ef733b828" Jul 28 20:19:35.121178 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.121154 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:35.121242 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.121225 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:35.121279 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:35.121257 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:19:36.128038 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:36.128004 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerStarted","Data":"5dbeb6740b22b11616d4767777972bdd19228a066575ce886fdb1ee7cff9edf9"} Jul 28 20:19:36.226729 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:36.226693 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:36.226892 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:36.226875 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 28 20:19:36.226940 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:36.226918 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 28 20:19:37.226768 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:37.226735 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 28 20:19:37.227230 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:37.226801 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 28 20:19:38.226826 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:38.226789 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 28 20:19:38.227310 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:38.226870 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 28 20:19:39.227068 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:39.227035 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 28 20:19:39.227471 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:39.227088 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 28 20:19:40.227101 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:40.227055 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 28 20:19:40.227576 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:40.227130 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 28 20:19:41.229637 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:41.229608 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:41.230105 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:41.229883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:19:41.230575 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:19:41.230550 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:21:21.332921 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:21:21.332836 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:21:21.344417 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:21:21.344390 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:21:21.538645 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:21:21.538614 2562 scope.go:117] "RemoveContainer" containerID="9110b6a6a0aab45ef2c07695005c4aa3ee1ad36090d5018055464bf393e427d1" Jul 28 20:21:21.548284 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:21:21.548264 2562 scope.go:117] "RemoveContainer" containerID="d02ddf09cdc247b32f865e658fd0ad212bc3cb39725a999db673d1f5161c6de8" Jul 28 20:23:21.584812 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:23:21.584723 2562 scope.go:117] "RemoveContainer" containerID="2715770bc2bfd420cfbb86e553bb6139b041a31d84e18d3baaa58804e3d4bea6" Jul 28 20:23:21.593778 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:23:21.593759 2562 scope.go:117] "RemoveContainer" containerID="250c3c91167b692224e630f99f578657bb740df8ba6eafc2a0559b8456955b3d" Jul 28 20:24:21.620349 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:24:21.620277 2562 scope.go:117] "RemoveContainer" containerID="825474a14ff982b6ac4d2fdfe4bfb1d29b879d216a76ecc2b7bb4777f9e95450" Jul 28 20:25:21.641962 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:25:21.641931 2562 scope.go:117] "RemoveContainer" containerID="f5c38074153782e925b32c08de8b4791315ad273d93db286c070262006724688" Jul 28 20:26:21.374009 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:26:21.373981 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:26:21.384422 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:26:21.384399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:28:29.446232 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.446202 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413217456"] Jul 28 20:28:29.449950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.449932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:29.453485 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.453467 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w9ms5\"" Jul 28 20:28:29.458338 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.458306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413217456"] Jul 28 20:28:29.544786 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.544761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vq9v\" (UniqueName: \"kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v\") pod \"test-curl-7352-139710413217456\" (UID: \"becbbb8e-a5a2-400c-80fd-85b35156dffc\") " pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:29.646230 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.646200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq9v\" (UniqueName: \"kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v\") pod \"test-curl-7352-139710413217456\" (UID: \"becbbb8e-a5a2-400c-80fd-85b35156dffc\") " pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:29.656400 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.656374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq9v\" (UniqueName: \"kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v\") pod \"test-curl-7352-139710413217456\" (UID: \"becbbb8e-a5a2-400c-80fd-85b35156dffc\") " pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:29.760362 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:29.760290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:30.109760 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:30.109731 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413217456"] Jul 28 20:28:30.109926 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:30.109832 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbecbbb8e_a5a2_400c_80fd_85b35156dffc.slice/crio-a51f76a07f5b72f1ec4b154db0829b41f816e1bfe83042b8cfffaa6b71ed5c3f WatchSource:0}: Error finding container a51f76a07f5b72f1ec4b154db0829b41f816e1bfe83042b8cfffaa6b71ed5c3f: Status 404 returned error can't find the container with id a51f76a07f5b72f1ec4b154db0829b41f816e1bfe83042b8cfffaa6b71ed5c3f Jul 28 20:28:30.111351 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:30.111335 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:28:30.344324 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:30.344289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413217456" event={"ID":"becbbb8e-a5a2-400c-80fd-85b35156dffc","Type":"ContainerStarted","Data":"a51f76a07f5b72f1ec4b154db0829b41f816e1bfe83042b8cfffaa6b71ed5c3f"} Jul 28 20:28:31.351346 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:31.351311 2562 generic.go:356] "Generic (PLEG): container finished" podID="becbbb8e-a5a2-400c-80fd-85b35156dffc" containerID="251d5ca524087c3df11ad1f12853042ccb75d247be42108104e00fe314723336" exitCode=0 Jul 28 20:28:31.351811 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:31.351390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413217456" event={"ID":"becbbb8e-a5a2-400c-80fd-85b35156dffc","Type":"ContainerDied","Data":"251d5ca524087c3df11ad1f12853042ccb75d247be42108104e00fe314723336"} Jul 28 20:28:32.499105 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.499080 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:32.579860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.579830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vq9v\" (UniqueName: \"kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v\") pod \"becbbb8e-a5a2-400c-80fd-85b35156dffc\" (UID: \"becbbb8e-a5a2-400c-80fd-85b35156dffc\") " Jul 28 20:28:32.581884 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.581851 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v" (OuterVolumeSpecName: "kube-api-access-7vq9v") pod "becbbb8e-a5a2-400c-80fd-85b35156dffc" (UID: "becbbb8e-a5a2-400c-80fd-85b35156dffc"). InnerVolumeSpecName "kube-api-access-7vq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:32.610815 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.610738 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413217456_becbbb8e-a5a2-400c-80fd-85b35156dffc/test-curl-7352-139710413217456/0.log" Jul 28 20:28:32.655053 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.655024 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413217456"] Jul 28 20:28:32.664128 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.664104 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413217456"] Jul 28 20:28:32.681350 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.681330 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vq9v\" (UniqueName: \"kubernetes.io/projected/becbbb8e-a5a2-400c-80fd-85b35156dffc-kube-api-access-7vq9v\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:32.812776 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.812749 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413166384"] Jul 28 20:28:32.813196 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.813182 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="becbbb8e-a5a2-400c-80fd-85b35156dffc" containerName="test-curl-7352-139710413217456" Jul 28 20:28:32.813241 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.813198 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="becbbb8e-a5a2-400c-80fd-85b35156dffc" containerName="test-curl-7352-139710413217456" Jul 28 20:28:32.813284 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.813274 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="becbbb8e-a5a2-400c-80fd-85b35156dffc" containerName="test-curl-7352-139710413217456" Jul 28 20:28:32.816341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.816326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:32.828246 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.828223 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413166384"] Jul 28 20:28:32.884190 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.884121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grvf\" (UniqueName: \"kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf\") pod \"test-curl-7352-139710413166384\" (UID: \"da880785-06ca-4916-99d5-d9af61e5dda1\") " pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:32.984831 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.984799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grvf\" (UniqueName: \"kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf\") pod \"test-curl-7352-139710413166384\" (UID: \"da880785-06ca-4916-99d5-d9af61e5dda1\") " pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:32.996199 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:32.996178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grvf\" (UniqueName: \"kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf\") pod \"test-curl-7352-139710413166384\" (UID: \"da880785-06ca-4916-99d5-d9af61e5dda1\") " pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:33.125206 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.125181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:33.163482 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.163455 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="becbbb8e-a5a2-400c-80fd-85b35156dffc" path="/var/lib/kubelet/pods/becbbb8e-a5a2-400c-80fd-85b35156dffc/volumes" Jul 28 20:28:33.283785 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.283760 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413166384"] Jul 28 20:28:33.285290 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:33.285261 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda880785_06ca_4916_99d5_d9af61e5dda1.slice/crio-85d041457c4c8581e3dec1cdc5f7dd626569acc64c965283a62d70e681a03a82 WatchSource:0}: Error finding container 85d041457c4c8581e3dec1cdc5f7dd626569acc64c965283a62d70e681a03a82: Status 404 returned error can't find the container with id 85d041457c4c8581e3dec1cdc5f7dd626569acc64c965283a62d70e681a03a82 Jul 28 20:28:33.364044 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.364017 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413217456" Jul 28 20:28:33.364159 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.364075 2562 scope.go:117] "RemoveContainer" containerID="251d5ca524087c3df11ad1f12853042ccb75d247be42108104e00fe314723336" Jul 28 20:28:33.365381 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:33.365352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413166384" event={"ID":"da880785-06ca-4916-99d5-d9af61e5dda1","Type":"ContainerStarted","Data":"85d041457c4c8581e3dec1cdc5f7dd626569acc64c965283a62d70e681a03a82"} Jul 28 20:28:34.373016 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:34.372983 2562 generic.go:356] "Generic (PLEG): container finished" podID="da880785-06ca-4916-99d5-d9af61e5dda1" containerID="8171d7be3e864e24099ed57c9e58d850a71ac8a85e14ffbf3a6991ca4c4b549a" exitCode=0 Jul 28 20:28:34.373448 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:34.373074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413166384" event={"ID":"da880785-06ca-4916-99d5-d9af61e5dda1","Type":"ContainerDied","Data":"8171d7be3e864e24099ed57c9e58d850a71ac8a85e14ffbf3a6991ca4c4b549a"} Jul 28 20:28:35.518934 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.518912 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:35.540028 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.539990 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413166384_da880785-06ca-4916-99d5-d9af61e5dda1/test-curl-7352-139710413166384/0.log" Jul 28 20:28:35.562914 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.562888 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413166384"] Jul 28 20:28:35.566627 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.566605 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413166384"] Jul 28 20:28:35.609087 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.609061 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8grvf\" (UniqueName: \"kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf\") pod \"da880785-06ca-4916-99d5-d9af61e5dda1\" (UID: \"da880785-06ca-4916-99d5-d9af61e5dda1\") " Jul 28 20:28:35.611154 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.611093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf" (OuterVolumeSpecName: "kube-api-access-8grvf") pod "da880785-06ca-4916-99d5-d9af61e5dda1" (UID: "da880785-06ca-4916-99d5-d9af61e5dda1"). InnerVolumeSpecName "kube-api-access-8grvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:35.710297 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.710273 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grvf\" (UniqueName: \"kubernetes.io/projected/da880785-06ca-4916-99d5-d9af61e5dda1-kube-api-access-8grvf\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:35.740041 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.740017 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:35.740494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.740477 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da880785-06ca-4916-99d5-d9af61e5dda1" containerName="test-curl-7352-139710413166384" Jul 28 20:28:35.740579 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.740497 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da880785-06ca-4916-99d5-d9af61e5dda1" containerName="test-curl-7352-139710413166384" Jul 28 20:28:35.740636 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.740614 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da880785-06ca-4916-99d5-d9af61e5dda1" containerName="test-curl-7352-139710413166384" Jul 28 20:28:35.743902 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.743884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:35.749561 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.749537 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:35.811131 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.811100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2k9\" (UniqueName: \"kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9\") pod \"test-curl-7352-139710413101072\" (UID: \"60739562-3a87-4b75-bb8d-429687cbba6c\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:35.912114 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.912082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2k9\" (UniqueName: \"kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9\") pod \"test-curl-7352-139710413101072\" (UID: \"60739562-3a87-4b75-bb8d-429687cbba6c\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:35.921179 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:35.921148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2k9\" (UniqueName: \"kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9\") pod \"test-curl-7352-139710413101072\" (UID: \"60739562-3a87-4b75-bb8d-429687cbba6c\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:36.056631 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:36.056546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:36.197351 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:36.197323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:36.198214 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:36.198188 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60739562_3a87_4b75_bb8d_429687cbba6c.slice/crio-d515dcf5cf1bda01a4697230796bb44a955b97dbebf4a870394364bda2474dc3 WatchSource:0}: Error finding container d515dcf5cf1bda01a4697230796bb44a955b97dbebf4a870394364bda2474dc3: Status 404 returned error can't find the container with id d515dcf5cf1bda01a4697230796bb44a955b97dbebf4a870394364bda2474dc3 Jul 28 20:28:36.394002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:36.393910 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:36.394002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:36.393940 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d041457c4c8581e3dec1cdc5f7dd626569acc64c965283a62d70e681a03a82" Jul 28 20:28:36.395243 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:36.395214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413101072" event={"ID":"60739562-3a87-4b75-bb8d-429687cbba6c","Type":"ContainerStarted","Data":"d515dcf5cf1bda01a4697230796bb44a955b97dbebf4a870394364bda2474dc3"} Jul 28 20:28:36.405314 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:28:36.405288 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7352-139710413166384\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="da880785-06ca-4916-99d5-d9af61e5dda1" pod="opendatahub/test-curl-7352-139710413166384" Jul 28 20:28:37.163594 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:37.163551 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da880785-06ca-4916-99d5-d9af61e5dda1" path="/var/lib/kubelet/pods/da880785-06ca-4916-99d5-d9af61e5dda1/volumes" Jul 28 20:28:37.402988 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:37.402957 2562 generic.go:356] "Generic (PLEG): container finished" podID="60739562-3a87-4b75-bb8d-429687cbba6c" containerID="8b0cc08905e8ff67cb8bf77904216f1cb5e734844fe397d765e3fd1a78d7ba2a" exitCode=0 Jul 28 20:28:37.403186 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:37.403023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413101072" event={"ID":"60739562-3a87-4b75-bb8d-429687cbba6c","Type":"ContainerDied","Data":"8b0cc08905e8ff67cb8bf77904216f1cb5e734844fe397d765e3fd1a78d7ba2a"} Jul 28 20:28:38.548020 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.547990 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:38.573615 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.573587 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413101072_60739562-3a87-4b75-bb8d-429687cbba6c/test-curl-7352-139710413101072/0.log" Jul 28 20:28:38.597957 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.597927 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:38.603277 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.603255 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:38.635456 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.635433 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb2k9\" (UniqueName: \"kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9\") pod \"60739562-3a87-4b75-bb8d-429687cbba6c\" (UID: \"60739562-3a87-4b75-bb8d-429687cbba6c\") " Jul 28 20:28:38.637435 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.637412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9" (OuterVolumeSpecName: "kube-api-access-sb2k9") pod "60739562-3a87-4b75-bb8d-429687cbba6c" (UID: "60739562-3a87-4b75-bb8d-429687cbba6c"). InnerVolumeSpecName "kube-api-access-sb2k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:38.725491 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.725425 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:38.725852 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.725837 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60739562-3a87-4b75-bb8d-429687cbba6c" containerName="test-curl-7352-139710413101072" Jul 28 20:28:38.725908 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.725854 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60739562-3a87-4b75-bb8d-429687cbba6c" containerName="test-curl-7352-139710413101072" Jul 28 20:28:38.725949 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.725932 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60739562-3a87-4b75-bb8d-429687cbba6c" containerName="test-curl-7352-139710413101072" Jul 28 20:28:38.729022 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.729007 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:38.733917 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.733895 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:38.736436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.736413 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2k9\" (UniqueName: \"kubernetes.io/projected/60739562-3a87-4b75-bb8d-429687cbba6c-kube-api-access-sb2k9\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:38.837647 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.837614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvrk\" (UniqueName: \"kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk\") pod \"test-curl-7352-139710413101072\" (UID: \"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:38.938295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.938268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvrk\" (UniqueName: \"kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk\") pod \"test-curl-7352-139710413101072\" (UID: \"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:38.947522 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:38.947492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvrk\" (UniqueName: \"kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk\") pod \"test-curl-7352-139710413101072\" (UID: \"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f\") " pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:39.040304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.040233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:39.164700 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.164652 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60739562-3a87-4b75-bb8d-429687cbba6c" path="/var/lib/kubelet/pods/60739562-3a87-4b75-bb8d-429687cbba6c/volumes" Jul 28 20:28:39.215833 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.215806 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:39.216260 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:39.216235 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b6ab56_9ec0_4304_b0c1_cff2b918ce6f.slice/crio-13b98a8332238e86e0ba91bb59f3f772899460c67ded7f28d479a4230d922160 WatchSource:0}: Error finding container 13b98a8332238e86e0ba91bb59f3f772899460c67ded7f28d479a4230d922160: Status 404 returned error can't find the container with id 13b98a8332238e86e0ba91bb59f3f772899460c67ded7f28d479a4230d922160 Jul 28 20:28:39.415780 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.415754 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:39.415780 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.415781 2562 scope.go:117] "RemoveContainer" containerID="8b0cc08905e8ff67cb8bf77904216f1cb5e734844fe397d765e3fd1a78d7ba2a" Jul 28 20:28:39.417082 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:39.417057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413101072" event={"ID":"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f","Type":"ContainerStarted","Data":"13b98a8332238e86e0ba91bb59f3f772899460c67ded7f28d479a4230d922160"} Jul 28 20:28:40.425102 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:40.425070 2562 generic.go:356] "Generic (PLEG): container finished" podID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" containerID="cd1ab7c4c16c3d21dc936c2e384f33708295bcba0648315f767365a2f2352b43" exitCode=0 Jul 28 20:28:40.425489 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:40.425156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413101072" event={"ID":"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f","Type":"ContainerDied","Data":"cd1ab7c4c16c3d21dc936c2e384f33708295bcba0648315f767365a2f2352b43"} Jul 28 20:28:41.569787 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.569757 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:41.592391 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.592365 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413101072_a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f/test-curl-7352-139710413101072/0.log" Jul 28 20:28:41.620388 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.620357 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:41.624273 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.624246 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413101072"] Jul 28 20:28:41.663529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.663509 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvrk\" (UniqueName: \"kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk\") pod \"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f\" (UID: \"a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f\") " Jul 28 20:28:41.665687 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.665648 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk" (OuterVolumeSpecName: "kube-api-access-qjvrk") pod "a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" (UID: "a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f"). InnerVolumeSpecName "kube-api-access-qjvrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:41.747304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.747233 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:41.747694 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.747680 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" containerName="test-curl-7352-139710413101072" Jul 28 20:28:41.747752 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.747695 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" containerName="test-curl-7352-139710413101072" Jul 28 20:28:41.747791 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.747768 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" containerName="test-curl-7352-139710413101072" Jul 28 20:28:41.750727 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.750713 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:41.757269 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.757245 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:41.764695 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.764654 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvrk\" (UniqueName: \"kubernetes.io/projected/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f-kube-api-access-qjvrk\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:41.865706 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.865653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jp5\" (UniqueName: \"kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5\") pod \"test-curl-7352-139710413102080\" (UID: \"f798d0c1-07e6-4da4-987c-a98cc0f58c49\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:41.966832 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.966803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jp5\" (UniqueName: \"kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5\") pod \"test-curl-7352-139710413102080\" (UID: \"f798d0c1-07e6-4da4-987c-a98cc0f58c49\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:41.976944 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:41.976917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jp5\" (UniqueName: \"kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5\") pod \"test-curl-7352-139710413102080\" (UID: \"f798d0c1-07e6-4da4-987c-a98cc0f58c49\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:42.061748 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:42.061687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:42.206730 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:42.206701 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf798d0c1_07e6_4da4_987c_a98cc0f58c49.slice/crio-38d1f69202b7d9312664524a451b3c06c33fe0137f9821ec0c16b1e3e5011812 WatchSource:0}: Error finding container 38d1f69202b7d9312664524a451b3c06c33fe0137f9821ec0c16b1e3e5011812: Status 404 returned error can't find the container with id 38d1f69202b7d9312664524a451b3c06c33fe0137f9821ec0c16b1e3e5011812 Jul 28 20:28:42.206948 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:42.206924 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:42.438145 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:42.438117 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b98a8332238e86e0ba91bb59f3f772899460c67ded7f28d479a4230d922160" Jul 28 20:28:42.438145 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:42.438119 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:42.439339 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:42.439313 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413102080" event={"ID":"f798d0c1-07e6-4da4-987c-a98cc0f58c49","Type":"ContainerStarted","Data":"38d1f69202b7d9312664524a451b3c06c33fe0137f9821ec0c16b1e3e5011812"} Jul 28 20:28:42.449060 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:28:42.449037 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7352-139710413101072\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:43.081166 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.081126 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:43.085093 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.085072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:43.087699 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:28:43.087657 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7352-139710413101072\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" pod="opendatahub/test-curl-7352-139710413101072" Jul 28 20:28:43.094322 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.094216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:43.164293 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.164257 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f" path="/var/lib/kubelet/pods/a9b6ab56-9ec0-4304-b0c1-cff2b918ce6f/volumes" Jul 28 20:28:43.178461 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.178439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88lt\" (UniqueName: \"kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt\") pod \"kuadrant-operator-catalog-qzn8p\" (UID: \"1c63e8de-1f65-45a5-8206-8445df4fb462\") " pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:43.278979 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.278956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88lt\" (UniqueName: \"kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt\") pod \"kuadrant-operator-catalog-qzn8p\" (UID: \"1c63e8de-1f65-45a5-8206-8445df4fb462\") " pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:43.288399 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.288377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88lt\" (UniqueName: \"kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt\") pod \"kuadrant-operator-catalog-qzn8p\" (UID: \"1c63e8de-1f65-45a5-8206-8445df4fb462\") " pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:43.397506 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.397435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:43.451549 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.451501 2562 generic.go:356] "Generic (PLEG): container finished" podID="f798d0c1-07e6-4da4-987c-a98cc0f58c49" containerID="f0b6e5db5eea9d7534a2fcdec77cea2a3d0d93144138d8385a5b91dc2f57c9ab" exitCode=0 Jul 28 20:28:43.451823 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.451751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413102080" event={"ID":"f798d0c1-07e6-4da4-987c-a98cc0f58c49","Type":"ContainerDied","Data":"f0b6e5db5eea9d7534a2fcdec77cea2a3d0d93144138d8385a5b91dc2f57c9ab"} Jul 28 20:28:43.536043 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:43.536011 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:43.538055 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:43.538032 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c63e8de_1f65_45a5_8206_8445df4fb462.slice/crio-b04dd2cf6fcfc7bfe14e27f5be1b5392eb44c6f6fabad34c9a684445c429c30a WatchSource:0}: Error finding container b04dd2cf6fcfc7bfe14e27f5be1b5392eb44c6f6fabad34c9a684445c429c30a: Status 404 returned error can't find the container with id b04dd2cf6fcfc7bfe14e27f5be1b5392eb44c6f6fabad34c9a684445c429c30a Jul 28 20:28:44.462153 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.462108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" event={"ID":"1c63e8de-1f65-45a5-8206-8445df4fb462","Type":"ContainerStarted","Data":"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0"} Jul 28 20:28:44.462571 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.462169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" event={"ID":"1c63e8de-1f65-45a5-8206-8445df4fb462","Type":"ContainerStarted","Data":"b04dd2cf6fcfc7bfe14e27f5be1b5392eb44c6f6fabad34c9a684445c429c30a"} Jul 28 20:28:44.608423 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.608399 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:44.630873 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.630821 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" podStartSLOduration=1.263380508 podStartE2EDuration="1.630802093s" totalImagesPullingTime="367.421585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:44.494936197 +0000 UTC m=+3143.930890167" watchObservedRunningTime="2026-07-28 20:28:44.630802093 +0000 UTC m=+3144.066756045" Jul 28 20:28:44.638040 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.638021 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413102080_f798d0c1-07e6-4da4-987c-a98cc0f58c49/test-curl-7352-139710413102080/0.log" Jul 28 20:28:44.671469 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.671446 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:44.681050 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.681024 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:44.692646 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.692623 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8jp5\" (UniqueName: \"kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5\") pod \"f798d0c1-07e6-4da4-987c-a98cc0f58c49\" (UID: \"f798d0c1-07e6-4da4-987c-a98cc0f58c49\") " Jul 28 20:28:44.694704 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.694675 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5" (OuterVolumeSpecName: "kube-api-access-w8jp5") pod "f798d0c1-07e6-4da4-987c-a98cc0f58c49" (UID: "f798d0c1-07e6-4da4-987c-a98cc0f58c49"). InnerVolumeSpecName "kube-api-access-w8jp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:44.794369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.794307 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jp5\" (UniqueName: \"kubernetes.io/projected/f798d0c1-07e6-4da4-987c-a98cc0f58c49-kube-api-access-w8jp5\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:44.805124 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.805101 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:44.805508 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.805495 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f798d0c1-07e6-4da4-987c-a98cc0f58c49" containerName="test-curl-7352-139710413102080" Jul 28 20:28:44.805552 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.805509 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f798d0c1-07e6-4da4-987c-a98cc0f58c49" containerName="test-curl-7352-139710413102080" Jul 28 20:28:44.805597 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.805587 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f798d0c1-07e6-4da4-987c-a98cc0f58c49" containerName="test-curl-7352-139710413102080" Jul 28 20:28:44.808550 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.808535 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:44.818999 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.818553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:44.895213 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.895171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592c7\" (UniqueName: \"kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7\") pod \"test-curl-7352-139710413102080\" (UID: \"eaa07dc6-5d87-4cdd-9367-49d470921f54\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:44.996516 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:44.996486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592c7\" (UniqueName: \"kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7\") pod \"test-curl-7352-139710413102080\" (UID: \"eaa07dc6-5d87-4cdd-9367-49d470921f54\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:45.006259 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.006227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-592c7\" (UniqueName: \"kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7\") pod \"test-curl-7352-139710413102080\" (UID: \"eaa07dc6-5d87-4cdd-9367-49d470921f54\") " pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:45.130002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.129932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:45.164799 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.164770 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f798d0c1-07e6-4da4-987c-a98cc0f58c49" path="/var/lib/kubelet/pods/f798d0c1-07e6-4da4-987c-a98cc0f58c49/volumes" Jul 28 20:28:45.290032 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:28:45.290004 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaa07dc6_5d87_4cdd_9367_49d470921f54.slice/crio-4c83d2e3ab1e80442a7a1bc63171171408245694e37ab61f348bf14b83fad5a1 WatchSource:0}: Error finding container 4c83d2e3ab1e80442a7a1bc63171171408245694e37ab61f348bf14b83fad5a1: Status 404 returned error can't find the container with id 4c83d2e3ab1e80442a7a1bc63171171408245694e37ab61f348bf14b83fad5a1 Jul 28 20:28:45.290679 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.290628 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:45.469823 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.469798 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:45.470241 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.469803 2562 scope.go:117] "RemoveContainer" containerID="f0b6e5db5eea9d7534a2fcdec77cea2a3d0d93144138d8385a5b91dc2f57c9ab" Jul 28 20:28:45.471204 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:45.471176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413102080" event={"ID":"eaa07dc6-5d87-4cdd-9367-49d470921f54","Type":"ContainerStarted","Data":"4c83d2e3ab1e80442a7a1bc63171171408245694e37ab61f348bf14b83fad5a1"} Jul 28 20:28:46.478948 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:46.478916 2562 generic.go:356] "Generic (PLEG): container finished" podID="eaa07dc6-5d87-4cdd-9367-49d470921f54" containerID="e3cbfb03f170fcd079f2f1866cbd263caf0678646139ea52fb5a7ea64556c78f" exitCode=0 Jul 28 20:28:46.479299 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:46.479002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-139710413102080" event={"ID":"eaa07dc6-5d87-4cdd-9367-49d470921f54","Type":"ContainerDied","Data":"e3cbfb03f170fcd079f2f1866cbd263caf0678646139ea52fb5a7ea64556c78f"} Jul 28 20:28:47.633442 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.633421 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:47.655972 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.655951 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7352-139710413102080_eaa07dc6-5d87-4cdd-9367-49d470921f54/test-curl-7352-139710413102080/0.log" Jul 28 20:28:47.680210 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.680181 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:47.684945 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.684923 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-139710413102080"] Jul 28 20:28:47.722560 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.722540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-592c7\" (UniqueName: \"kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7\") pod \"eaa07dc6-5d87-4cdd-9367-49d470921f54\" (UID: \"eaa07dc6-5d87-4cdd-9367-49d470921f54\") " Jul 28 20:28:47.724710 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.724690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7" (OuterVolumeSpecName: "kube-api-access-592c7") pod "eaa07dc6-5d87-4cdd-9367-49d470921f54" (UID: "eaa07dc6-5d87-4cdd-9367-49d470921f54"). InnerVolumeSpecName "kube-api-access-592c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:47.823851 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:47.823793 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592c7\" (UniqueName: \"kubernetes.io/projected/eaa07dc6-5d87-4cdd-9367-49d470921f54-kube-api-access-592c7\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:48.492209 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:48.492178 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:48.492209 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:48.492199 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c83d2e3ab1e80442a7a1bc63171171408245694e37ab61f348bf14b83fad5a1" Jul 28 20:28:48.504232 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:28:48.504179 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7352-139710413102080\" is forbidden: User \"system:node:ip-10-0-132-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-248.ec2.internal' and this object" podUID="eaa07dc6-5d87-4cdd-9367-49d470921f54" pod="opendatahub/test-curl-7352-139710413102080" Jul 28 20:28:49.163704 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:49.163644 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa07dc6-5d87-4cdd-9367-49d470921f54" path="/var/lib/kubelet/pods/eaa07dc6-5d87-4cdd-9367-49d470921f54/volumes" Jul 28 20:28:53.398526 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:53.398445 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:53.398526 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:53.398501 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:53.419802 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:53.419780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:53.541248 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:53.541225 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:53.686557 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:53.686524 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:55.531963 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:55.531924 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" podUID="1c63e8de-1f65-45a5-8206-8445df4fb462" containerName="registry-server" containerID="cri-o://bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0" gracePeriod=2 Jul 28 20:28:55.794546 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:55.794470 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:55.898743 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:55.898714 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p88lt\" (UniqueName: \"kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt\") pod \"1c63e8de-1f65-45a5-8206-8445df4fb462\" (UID: \"1c63e8de-1f65-45a5-8206-8445df4fb462\") " Jul 28 20:28:55.900724 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:55.900693 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt" (OuterVolumeSpecName: "kube-api-access-p88lt") pod "1c63e8de-1f65-45a5-8206-8445df4fb462" (UID: "1c63e8de-1f65-45a5-8206-8445df4fb462"). InnerVolumeSpecName "kube-api-access-p88lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:28:55.999403 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:55.999373 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88lt\" (UniqueName: \"kubernetes.io/projected/1c63e8de-1f65-45a5-8206-8445df4fb462-kube-api-access-p88lt\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:28:56.538906 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.538870 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c63e8de-1f65-45a5-8206-8445df4fb462" containerID="bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0" exitCode=0 Jul 28 20:28:56.539369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.538913 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" Jul 28 20:28:56.539369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.538925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" event={"ID":"1c63e8de-1f65-45a5-8206-8445df4fb462","Type":"ContainerDied","Data":"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0"} Jul 28 20:28:56.539369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.538976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qzn8p" event={"ID":"1c63e8de-1f65-45a5-8206-8445df4fb462","Type":"ContainerDied","Data":"b04dd2cf6fcfc7bfe14e27f5be1b5392eb44c6f6fabad34c9a684445c429c30a"} Jul 28 20:28:56.539369 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.539003 2562 scope.go:117] "RemoveContainer" containerID="bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0" Jul 28 20:28:56.549277 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.549260 2562 scope.go:117] "RemoveContainer" containerID="bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0" Jul 28 20:28:56.549868 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:28:56.549841 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0\": container with ID starting with bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0 not found: ID does not exist" containerID="bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0" Jul 28 20:28:56.549943 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.549880 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0"} err="failed to get container status \"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0\": rpc error: code = NotFound desc = could not find container \"bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0\": container with ID starting with bca275eff7d5da59c3274d6fae572d46469cb0255d5d1264121687e3279c66d0 not found: ID does not exist" Jul 28 20:28:56.565843 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.565819 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:56.582642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:56.575711 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qzn8p"] Jul 28 20:28:57.163098 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:28:57.163068 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c63e8de-1f65-45a5-8206-8445df4fb462" path="/var/lib/kubelet/pods/1c63e8de-1f65-45a5-8206-8445df4fb462/volumes" Jul 28 20:29:57.553827 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:29:57.553797 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:30:00.277487 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.277450 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj"] Jul 28 20:30:00.277950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.277935 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa07dc6-5d87-4cdd-9367-49d470921f54" containerName="test-curl-7352-139710413102080" Jul 28 20:30:00.277950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.277951 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa07dc6-5d87-4cdd-9367-49d470921f54" containerName="test-curl-7352-139710413102080" Jul 28 20:30:00.278027 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.277966 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c63e8de-1f65-45a5-8206-8445df4fb462" containerName="registry-server" Jul 28 20:30:00.278027 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.277972 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c63e8de-1f65-45a5-8206-8445df4fb462" containerName="registry-server" Jul 28 20:30:00.278100 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.278041 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c63e8de-1f65-45a5-8206-8445df4fb462" containerName="registry-server" Jul 28 20:30:00.278100 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.278050 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa07dc6-5d87-4cdd-9367-49d470921f54" containerName="test-curl-7352-139710413102080" Jul 28 20:30:00.281196 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.281179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" Jul 28 20:30:00.302307 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.302280 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:30:00.304336 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.304318 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59"] Jul 28 20:30:00.307817 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.307798 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w"] Jul 28 20:30:00.307920 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.307900 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" Jul 28 20:30:00.311252 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.311233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" Jul 28 20:30:00.311349 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.311265 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" Jul 28 20:30:00.316219 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.316198 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7"] Jul 28 20:30:00.319182 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.319163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" Jul 28 20:30:00.319843 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.319825 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" Jul 28 20:30:00.341410 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.333452 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9"] Jul 28 20:30:00.344991 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.344710 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj"] Jul 28 20:30:00.344991 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.344767 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59"] Jul 28 20:30:00.344991 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.344871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" Jul 28 20:30:00.357930 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.357894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9"] Jul 28 20:30:00.371685 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.366904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" Jul 28 20:30:00.380746 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.378411 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7"] Jul 28 20:30:00.382628 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.382433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" Jul 28 20:30:00.384590 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.384500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w"] Jul 28 20:30:00.393608 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.393589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" Jul 28 20:30:00.659096 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.659063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7"] Jul 28 20:30:00.660199 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:00.660170 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623e09fd_6429_4c19_be7f_ebd5f86119ea.slice/crio-41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47 WatchSource:0}: Error finding container 41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47: Status 404 returned error can't find the container with id 41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47 Jul 28 20:30:00.678082 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.678054 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w"] Jul 28 20:30:00.683914 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:00.683886 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed8198f_5194_4d03_b87a_6d0df004070d.slice/crio-4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2 WatchSource:0}: Error finding container 4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2: Status 404 returned error can't find the container with id 4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2 Jul 28 20:30:00.701136 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.701109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9"] Jul 28 20:30:00.702194 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:00.702169 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4f444_18eb_4b6d_bab9_75fe4b3d127f.slice/crio-0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152 WatchSource:0}: Error finding container 0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152: Status 404 returned error can't find the container with id 0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152 Jul 28 20:30:00.745010 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:00.744983 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe577848_eb89_449f_a107_c1dc3a700b6b.slice/crio-9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594 WatchSource:0}: Error finding container 9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594: Status 404 returned error can't find the container with id 9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594 Jul 28 20:30:00.746067 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.746043 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59"] Jul 28 20:30:00.799319 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.799294 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj"] Jul 28 20:30:00.930107 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.930071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" event={"ID":"be577848-eb89-449f-a107-c1dc3a700b6b","Type":"ContainerStarted","Data":"9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594"} Jul 28 20:30:00.931347 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.931322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" event={"ID":"b41d5476-aba9-42df-ac10-351c8f908af2","Type":"ContainerStarted","Data":"e1db018191550cace74f4458414246110d2b669bc951c2f04512f5f8ec62da48"} Jul 28 20:30:00.932478 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.932457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" event={"ID":"6ed8198f-5194-4d03-b87a-6d0df004070d","Type":"ContainerStarted","Data":"4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2"} Jul 28 20:30:00.933553 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.933530 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" event={"ID":"623e09fd-6429-4c19-be7f-ebd5f86119ea","Type":"ContainerStarted","Data":"41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47"} Jul 28 20:30:00.934645 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:00.934624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" event={"ID":"3bf4f444-18eb-4b6d-bab9-75fe4b3d127f","Type":"ContainerStarted","Data":"0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152"} Jul 28 20:30:01.944274 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.944068 2562 generic.go:356] "Generic (PLEG): container finished" podID="623e09fd-6429-4c19-be7f-ebd5f86119ea" containerID="fb959c6e24b5741f695fffd91780df9cacf7876823b61b68ab6fcfad5f81317e" exitCode=0 Jul 28 20:30:01.944274 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.944183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" event={"ID":"623e09fd-6429-4c19-be7f-ebd5f86119ea","Type":"ContainerDied","Data":"fb959c6e24b5741f695fffd91780df9cacf7876823b61b68ab6fcfad5f81317e"} Jul 28 20:30:01.946696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.946546 2562 generic.go:356] "Generic (PLEG): container finished" podID="3bf4f444-18eb-4b6d-bab9-75fe4b3d127f" containerID="da7504a349fac05711f806a3f027fd2f824cdd8d04274cf3a24bd2116d2b3f12" exitCode=0 Jul 28 20:30:01.946696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.946646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" event={"ID":"3bf4f444-18eb-4b6d-bab9-75fe4b3d127f","Type":"ContainerDied","Data":"da7504a349fac05711f806a3f027fd2f824cdd8d04274cf3a24bd2116d2b3f12"} Jul 28 20:30:01.949344 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.949317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" event={"ID":"be577848-eb89-449f-a107-c1dc3a700b6b","Type":"ContainerStarted","Data":"5c222f90e23f636f8c9289e42d50488f0fb3d5cd55e5dc7dc5b1d2c55b63ba64"} Jul 28 20:30:01.951731 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.951690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" event={"ID":"b41d5476-aba9-42df-ac10-351c8f908af2","Type":"ContainerStarted","Data":"0ed527e9cc15963225a893d29bfd885ad6952207b898f06e4c86dfdd62184f44"} Jul 28 20:30:01.953402 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:01.953384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" event={"ID":"6ed8198f-5194-4d03-b87a-6d0df004070d","Type":"ContainerStarted","Data":"6377b61acf9fe3eb6e04ce42a60449ef9e184bbc309d38203a9c5e7a9429002e"} Jul 28 20:30:02.016656 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.016610 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" podStartSLOduration=2.016594413 podStartE2EDuration="2.016594413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:02.014847088 +0000 UTC m=+3221.450801043" watchObservedRunningTime="2026-07-28 20:30:02.016594413 +0000 UTC m=+3221.452548365" Jul 28 20:30:02.065456 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.065414 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" podStartSLOduration=2.065400899 podStartE2EDuration="2.065400899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:02.065372709 +0000 UTC m=+3221.501326662" watchObservedRunningTime="2026-07-28 20:30:02.065400899 +0000 UTC m=+3221.501354851" Jul 28 20:30:02.961473 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.961436 2562 generic.go:356] "Generic (PLEG): container finished" podID="be577848-eb89-449f-a107-c1dc3a700b6b" containerID="5c222f90e23f636f8c9289e42d50488f0fb3d5cd55e5dc7dc5b1d2c55b63ba64" exitCode=0 Jul 28 20:30:02.961927 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.961528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" event={"ID":"be577848-eb89-449f-a107-c1dc3a700b6b","Type":"ContainerDied","Data":"5c222f90e23f636f8c9289e42d50488f0fb3d5cd55e5dc7dc5b1d2c55b63ba64"} Jul 28 20:30:02.963086 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.963066 2562 generic.go:356] "Generic (PLEG): container finished" podID="b41d5476-aba9-42df-ac10-351c8f908af2" containerID="0ed527e9cc15963225a893d29bfd885ad6952207b898f06e4c86dfdd62184f44" exitCode=0 Jul 28 20:30:02.963193 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.963094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" event={"ID":"b41d5476-aba9-42df-ac10-351c8f908af2","Type":"ContainerDied","Data":"0ed527e9cc15963225a893d29bfd885ad6952207b898f06e4c86dfdd62184f44"} Jul 28 20:30:02.964639 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.964619 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ed8198f-5194-4d03-b87a-6d0df004070d" containerID="6377b61acf9fe3eb6e04ce42a60449ef9e184bbc309d38203a9c5e7a9429002e" exitCode=0 Jul 28 20:30:02.964758 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:02.964690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" event={"ID":"6ed8198f-5194-4d03-b87a-6d0df004070d","Type":"ContainerDied","Data":"6377b61acf9fe3eb6e04ce42a60449ef9e184bbc309d38203a9c5e7a9429002e"} Jul 28 20:30:03.119213 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.119185 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" Jul 28 20:30:03.157744 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.157717 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" Jul 28 20:30:03.168943 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.168916 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" Jul 28 20:30:03.971733 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.971706 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" Jul 28 20:30:03.972176 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.971735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w" event={"ID":"6ed8198f-5194-4d03-b87a-6d0df004070d","Type":"ContainerDied","Data":"4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2"} Jul 28 20:30:03.972176 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.971788 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c0823468c521d1d3c531756ed2fd2d1c9a7b3c3254427d31e8f76020e9f6bd2" Jul 28 20:30:03.973115 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.973096 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" event={"ID":"623e09fd-6429-4c19-be7f-ebd5f86119ea","Type":"ContainerDied","Data":"41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47"} Jul 28 20:30:03.973199 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.973143 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41aaca911c28e602bf8ee95672a07f031ba7c5a9442a3a673540c06715211f47" Jul 28 20:30:03.973199 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.973101 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7" Jul 28 20:30:03.974719 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.974693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" event={"ID":"3bf4f444-18eb-4b6d-bab9-75fe4b3d127f","Type":"ContainerDied","Data":"0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152"} Jul 28 20:30:03.974719 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.974704 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9" Jul 28 20:30:03.974910 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:03.974729 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fb5739aa1f7c874764ecd644c4b3f9f44b755a02a65a879241e7b5eca0fa152" Jul 28 20:30:04.136924 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.136897 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" Jul 28 20:30:04.139790 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.139770 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" Jul 28 20:30:04.982537 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.982511 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" Jul 28 20:30:04.982969 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.982547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59" event={"ID":"be577848-eb89-449f-a107-c1dc3a700b6b","Type":"ContainerDied","Data":"9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594"} Jul 28 20:30:04.982969 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.982595 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9118e49554f59111a7a65af55df10d8669e5f6fbff7dcdd0c508434a84277594" Jul 28 20:30:04.983992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.983977 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" Jul 28 20:30:04.984081 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.984022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-cslgj" event={"ID":"b41d5476-aba9-42df-ac10-351c8f908af2","Type":"ContainerDied","Data":"e1db018191550cace74f4458414246110d2b669bc951c2f04512f5f8ec62da48"} Jul 28 20:30:04.984081 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:04.984057 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1db018191550cace74f4458414246110d2b669bc951c2f04512f5f8ec62da48" Jul 28 20:30:54.980561 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:54.980533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:30:55.687147 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687113 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:30:55.687592 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687578 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed8198f-5194-4d03-b87a-6d0df004070d" containerName="cleanup" Jul 28 20:30:55.687640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687594 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed8198f-5194-4d03-b87a-6d0df004070d" containerName="cleanup" Jul 28 20:30:55.687640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687607 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf4f444-18eb-4b6d-bab9-75fe4b3d127f" containerName="cleanup" Jul 28 20:30:55.687640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687616 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf4f444-18eb-4b6d-bab9-75fe4b3d127f" containerName="cleanup" Jul 28 20:30:55.687640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687631 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41d5476-aba9-42df-ac10-351c8f908af2" containerName="cleanup" Jul 28 20:30:55.687640 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687636 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41d5476-aba9-42df-ac10-351c8f908af2" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687647 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623e09fd-6429-4c19-be7f-ebd5f86119ea" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687652 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="623e09fd-6429-4c19-be7f-ebd5f86119ea" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687670 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be577848-eb89-449f-a107-c1dc3a700b6b" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687675 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be577848-eb89-449f-a107-c1dc3a700b6b" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687752 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf4f444-18eb-4b6d-bab9-75fe4b3d127f" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687762 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed8198f-5194-4d03-b87a-6d0df004070d" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687770 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be577848-eb89-449f-a107-c1dc3a700b6b" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687776 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="623e09fd-6429-4c19-be7f-ebd5f86119ea" containerName="cleanup" Jul 28 20:30:55.687821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.687783 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41d5476-aba9-42df-ac10-351c8f908af2" containerName="cleanup" Jul 28 20:30:55.690926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.690911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.697679 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.697601 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e5feff-openshift-default-dockercfg-bxfgt\"" Jul 28 20:30:55.709559 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.709530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:30:55.794405 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794405 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794653 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnk2k\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794867 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.794867 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.794820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.895815 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.895782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.895992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.895824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.895992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.895868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896082 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896082 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896270 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896270 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896476 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896476 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnk2k\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896476 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896476 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896732 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.896944 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896923 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.897017 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.896992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.898211 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.898187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.898313 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.898271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.905834 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.905813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:55.906204 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:55.906183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnk2k\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k\") pod \"e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:56.001416 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.001327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:56.162798 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.162763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:30:56.166418 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:56.166396 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88211e45_a9ad_44d9_b6e6_49366c91d7c8.slice/crio-a92d03cb443f232f29eb1d58d56cc227572ffe60bcc28b9efa0d779b59da92b6 WatchSource:0}: Error finding container a92d03cb443f232f29eb1d58d56cc227572ffe60bcc28b9efa0d779b59da92b6: Status 404 returned error can't find the container with id a92d03cb443f232f29eb1d58d56cc227572ffe60bcc28b9efa0d779b59da92b6 Jul 28 20:30:56.168822 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.168789 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:30:56.168905 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.168869 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:30:56.168954 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.168906 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:30:56.304490 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.304394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerStarted","Data":"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946"} Jul 28 20:30:56.304490 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.304463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerStarted","Data":"a92d03cb443f232f29eb1d58d56cc227572ffe60bcc28b9efa0d779b59da92b6"} Jul 28 20:30:56.329447 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:56.329357 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podStartSLOduration=1.3293377849999999 podStartE2EDuration="1.329337785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:56.328642881 +0000 UTC m=+3275.764596859" watchObservedRunningTime="2026-07-28 20:30:56.329337785 +0000 UTC m=+3275.765291757" Jul 28 20:30:57.001614 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:57.001575 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:57.007078 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:57.007052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:57.311890 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:57.311789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:57.312745 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:57.312723 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:30:58.504047 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.504015 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:30:58.507923 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.507900 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.510652 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.510632 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e5feff-serving-cert\"" Jul 28 20:30:58.531805 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.531778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:30:58.623436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.623402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.623614 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.623502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.623614 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.623602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696h8\" (UniqueName: \"kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.724803 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.724766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696h8\" (UniqueName: \"kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.724974 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.724856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.724974 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.724883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.725054 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:30:58.724995 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-serving-cert: secret "maas-api-e2e-ipp-cleanup-e5feff-serving-cert" not found Jul 28 20:30:58.725088 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:30:58.725066 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls podName:35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d nodeName:}" failed. No retries permitted until 2026-07-28 20:30:59.225045769 +0000 UTC m=+3278.660999705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" (UID: "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d") : secret "maas-api-e2e-ipp-cleanup-e5feff-serving-cert" not found Jul 28 20:30:58.725486 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.725466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:58.743131 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:58.743105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696h8\" (UniqueName: \"kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:59.230509 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:59.230468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:59.232915 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:59.232894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:59.418791 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:59.418757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:30:59.788985 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:30:59.788954 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:30:59.790373 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:30:59.790346 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b227e1_e6f8_4e5f_8f86_3f8d3c53d91d.slice/crio-b0cf0cf0ad6efd3f27c4f8a125ed3a602fad14e7a01558912a6fa939502c4324 WatchSource:0}: Error finding container b0cf0cf0ad6efd3f27c4f8a125ed3a602fad14e7a01558912a6fa939502c4324: Status 404 returned error can't find the container with id b0cf0cf0ad6efd3f27c4f8a125ed3a602fad14e7a01558912a6fa939502c4324 Jul 28 20:31:00.333912 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:00.333878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" event={"ID":"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d","Type":"ContainerStarted","Data":"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed"} Jul 28 20:31:00.334047 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:00.333940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" event={"ID":"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d","Type":"ContainerStarted","Data":"b0cf0cf0ad6efd3f27c4f8a125ed3a602fad14e7a01558912a6fa939502c4324"} Jul 28 20:31:00.334139 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:00.334121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:31:00.364527 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:00.364483 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" podStartSLOduration=1.896626462 podStartE2EDuration="2.364466783s" totalImagesPullingTime="467.840321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:31:00.361348386 +0000 UTC m=+3279.797302337" watchObservedRunningTime="2026-07-28 20:31:00.364466783 +0000 UTC m=+3279.800420770" Jul 28 20:31:06.352090 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:06.352049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:31:11.414610 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.414579 2562 generic.go:356] "Generic (PLEG): container finished" podID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerID="c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946" exitCode=0 Jul 28 20:31:11.415085 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.414651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerDied","Data":"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946"} Jul 28 20:31:11.415155 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.415094 2562 scope.go:117] "RemoveContainer" containerID="c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946" Jul 28 20:31:11.416223 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.416197 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:31:11.416304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.416264 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:31:11.416304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:11.416295 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:31:12.311940 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:12.311910 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:12.422657 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:12.422621 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerStarted","Data":"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74"} Jul 28 20:31:13.001683 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:13.001629 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:13.003041 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:13.003013 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Jul 28 20:31:13.003186 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:13.003061 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Jul 28 20:31:14.002570 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:14.002536 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Jul 28 20:31:14.002992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:14.002590 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Jul 28 20:31:15.002002 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:15.001966 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Jul 28 20:31:15.002185 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:15.002038 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Jul 28 20:31:16.002022 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:16.001984 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Jul 28 20:31:16.002430 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:16.002052 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Jul 28 20:31:17.006114 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:17.006080 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:17.006585 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:17.006357 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:17.007144 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:17.007126 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:21.418684 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:21.418632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:31:21.430581 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:21.430551 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:31:24.090247 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.090216 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l"] Jul 28 20:31:24.093150 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.093132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:24.097291 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.097266 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:31:24.115613 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.115576 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l"] Jul 28 20:31:24.180191 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.180148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l\" (UID: \"09bc906d-65c2-498d-b04b-0f92d94d8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:24.281391 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.281285 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l\" (UID: \"09bc906d-65c2-498d-b04b-0f92d94d8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:24.282227 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.282196 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l\" (UID: \"09bc906d-65c2-498d-b04b-0f92d94d8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:24.404059 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.404024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:24.593473 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:24.593439 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l"] Jul 28 20:31:24.594161 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:31:24.594136 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bc906d_65c2_498d_b04b_0f92d94d8ba4.slice/crio-7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370 WatchSource:0}: Error finding container 7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370: Status 404 returned error can't find the container with id 7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370 Jul 28 20:31:25.513825 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:25.513787 2562 generic.go:356] "Generic (PLEG): container finished" podID="09bc906d-65c2-498d-b04b-0f92d94d8ba4" containerID="ab88662d685e3fa0373803ba5c8f9cc34b7d97e58faf57669e9a4c2aab881664" exitCode=0 Jul 28 20:31:25.514253 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:25.513871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" event={"ID":"09bc906d-65c2-498d-b04b-0f92d94d8ba4","Type":"ContainerDied","Data":"ab88662d685e3fa0373803ba5c8f9cc34b7d97e58faf57669e9a4c2aab881664"} Jul 28 20:31:25.514253 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:25.513934 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" event={"ID":"09bc906d-65c2-498d-b04b-0f92d94d8ba4","Type":"ContainerStarted","Data":"7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370"} Jul 28 20:31:26.667472 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:26.667448 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:26.702886 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:26.702845 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca\") pod \"09bc906d-65c2-498d-b04b-0f92d94d8ba4\" (UID: \"09bc906d-65c2-498d-b04b-0f92d94d8ba4\") " Jul 28 20:31:26.703300 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:26.703268 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09bc906d-65c2-498d-b04b-0f92d94d8ba4" (UID: "09bc906d-65c2-498d-b04b-0f92d94d8ba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:31:26.803809 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:26.803725 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc906d-65c2-498d-b04b-0f92d94d8ba4-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:27.529102 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:27.529020 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" Jul 28 20:31:27.529102 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:27.529037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l" event={"ID":"09bc906d-65c2-498d-b04b-0f92d94d8ba4","Type":"ContainerDied","Data":"7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370"} Jul 28 20:31:27.529102 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:27.529079 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6100f3d0220e79613f7246e5256cca206682567f0e8ecc49dc1f6ceea4e370" Jul 28 20:31:29.208784 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.208746 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:31:29.209279 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.208991 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" podUID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" containerName="maas-api" containerID="cri-o://ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed" gracePeriod=30 Jul 28 20:31:29.503011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.502984 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:31:29.519838 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.519796 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l"] Jul 28 20:31:29.522594 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.522568 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e5feff-pg55l"] Jul 28 20:31:29.531855 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.531833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-696h8\" (UniqueName: \"kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8\") pod \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " Jul 28 20:31:29.531968 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.531892 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") pod \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " Jul 28 20:31:29.531968 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.531930 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca\") pod \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\" (UID: \"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d\") " Jul 28 20:31:29.532450 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.532423 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" (UID: "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:31:29.534253 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.534231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" (UID: "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:31:29.534793 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.534765 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8" (OuterVolumeSpecName: "kube-api-access-696h8") pod "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" (UID: "35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d"). InnerVolumeSpecName "kube-api-access-696h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:31:29.555922 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.555871 2562 generic.go:356] "Generic (PLEG): container finished" podID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" containerID="ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed" exitCode=0 Jul 28 20:31:29.556068 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.555965 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" Jul 28 20:31:29.556068 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.555970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" event={"ID":"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d","Type":"ContainerDied","Data":"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed"} Jul 28 20:31:29.556189 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.556076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd" event={"ID":"35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d","Type":"ContainerDied","Data":"b0cf0cf0ad6efd3f27c4f8a125ed3a602fad14e7a01558912a6fa939502c4324"} Jul 28 20:31:29.556189 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.556107 2562 scope.go:117] "RemoveContainer" containerID="ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed" Jul 28 20:31:29.567282 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.567156 2562 scope.go:117] "RemoveContainer" containerID="ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed" Jul 28 20:31:29.567509 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:31:29.567483 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed\": container with ID starting with ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed not found: ID does not exist" containerID="ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed" Jul 28 20:31:29.567630 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.567512 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed"} err="failed to get container status \"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed\": rpc error: code = NotFound desc = could not find container \"ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed\": container with ID starting with ab61573003fb4785f4d1a96a87a9aaeddf7bf5cee10f82b39867785fcc5d0bed not found: ID does not exist" Jul 28 20:31:29.587870 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.587844 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:31:29.590575 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.590549 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e5feff-56fc9b9f5c-p47cd"] Jul 28 20:31:29.633574 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.633538 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696h8\" (UniqueName: \"kubernetes.io/projected/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-kube-api-access-696h8\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:29.633574 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.633566 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-maas-api-tls\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:29.633574 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:29.633577 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d-openshift-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:31.163306 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:31.163273 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09bc906d-65c2-498d-b04b-0f92d94d8ba4" path="/var/lib/kubelet/pods/09bc906d-65c2-498d-b04b-0f92d94d8ba4/volumes" Jul 28 20:31:31.163699 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:31.163675 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" path="/var/lib/kubelet/pods/35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d/volumes" Jul 28 20:31:37.337143 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:37.337109 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:31:37.337550 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:37.337315 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" containerID="cri-o://0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74" gracePeriod=30 Jul 28 20:31:41.361380 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361338 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb"] Jul 28 20:31:41.361879 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361860 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" containerName="maas-api" Jul 28 20:31:41.361936 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361881 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" containerName="maas-api" Jul 28 20:31:41.361936 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361909 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09bc906d-65c2-498d-b04b-0f92d94d8ba4" containerName="revoke-keys" Jul 28 20:31:41.361936 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361915 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09bc906d-65c2-498d-b04b-0f92d94d8ba4" containerName="revoke-keys" Jul 28 20:31:41.362040 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361977 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="09bc906d-65c2-498d-b04b-0f92d94d8ba4" containerName="revoke-keys" Jul 28 20:31:41.362040 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.361984 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b227e1-e6f8-4e5f-8f86-3f8d3c53d91d" containerName="maas-api" Jul 28 20:31:41.365382 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.365362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:41.370019 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.370000 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:31:41.384631 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.384597 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb"] Jul 28 20:31:41.450554 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.450511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb\" (UID: \"b25743dd-31b2-46e6-8738-e2f16b560c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:41.551643 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.551605 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb\" (UID: \"b25743dd-31b2-46e6-8738-e2f16b560c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:41.552261 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.552242 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb\" (UID: \"b25743dd-31b2-46e6-8738-e2f16b560c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:41.675479 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:41.675443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:42.041991 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:31:42.041940 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25743dd_31b2_46e6_8738_e2f16b560c4b.slice/crio-306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958 WatchSource:0}: Error finding container 306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958: Status 404 returned error can't find the container with id 306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958 Jul 28 20:31:42.042736 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.042709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb"] Jul 28 20:31:42.594515 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.594488 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:42.642354 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.642255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" event={"ID":"b25743dd-31b2-46e6-8738-e2f16b560c4b","Type":"ContainerStarted","Data":"ef4a2f114dc04918c46ad6bd42e8f1c2c9726052a3c3eedaf2f40339a7d9a3d5"} Jul 28 20:31:42.642354 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.642319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" event={"ID":"b25743dd-31b2-46e6-8738-e2f16b560c4b","Type":"ContainerStarted","Data":"306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958"} Jul 28 20:31:42.644913 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.644883 2562 generic.go:356] "Generic (PLEG): container finished" podID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerID="0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74" exitCode=0 Jul 28 20:31:42.645084 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.644949 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" Jul 28 20:31:42.645084 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.644973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerDied","Data":"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74"} Jul 28 20:31:42.645084 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.645017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx" event={"ID":"88211e45-a9ad-44d9-b6e6-49366c91d7c8","Type":"ContainerDied","Data":"a92d03cb443f232f29eb1d58d56cc227572ffe60bcc28b9efa0d779b59da92b6"} Jul 28 20:31:42.645084 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.645046 2562 scope.go:117] "RemoveContainer" containerID="0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74" Jul 28 20:31:42.656046 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.656025 2562 scope.go:117] "RemoveContainer" containerID="c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946" Jul 28 20:31:42.661565 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661530 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661728 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661574 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661728 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661619 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661728 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661691 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661912 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661912 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661912 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661819 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnk2k\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.661912 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661879 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.662123 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.661933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert\") pod \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\" (UID: \"88211e45-a9ad-44d9-b6e6-49366c91d7c8\") " Jul 28 20:31:42.662286 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.662246 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:31:42.662476 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.662433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:31:42.662776 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.662716 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:31:42.663696 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.663176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:31:42.664956 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.664926 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:31:42.665175 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.665141 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:31:42.665405 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.665364 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:31:42.666819 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.666791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k" (OuterVolumeSpecName: "kube-api-access-nnk2k") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "kube-api-access-nnk2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:31:42.668328 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.668303 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88211e45-a9ad-44d9-b6e6-49366c91d7c8" (UID: "88211e45-a9ad-44d9-b6e6-49366c91d7c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:31:42.670703 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.670682 2562 scope.go:117] "RemoveContainer" containerID="0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74" Jul 28 20:31:42.671033 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:31:42.671012 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74\": container with ID starting with 0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74 not found: ID does not exist" containerID="0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74" Jul 28 20:31:42.671115 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.671046 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74"} err="failed to get container status \"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74\": rpc error: code = NotFound desc = could not find container \"0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74\": container with ID starting with 0a75060513f05e9a4d51d2d1b9f26dcba4166cfae6f5e59c2e168aedca7d8e74 not found: ID does not exist" Jul 28 20:31:42.671115 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.671071 2562 scope.go:117] "RemoveContainer" containerID="c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946" Jul 28 20:31:42.671320 ip-10-0-132-248 kubenswrapper[2562]: E0728 20:31:42.671294 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946\": container with ID starting with c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946 not found: ID does not exist" containerID="c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946" Jul 28 20:31:42.671421 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.671323 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946"} err="failed to get container status \"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946\": rpc error: code = NotFound desc = could not find container \"c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946\": container with ID starting with c056108d50279187c83608a248f48de319364072d5d472017a404321a88f2946 not found: ID does not exist" Jul 28 20:31:42.676057 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.676012 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" podStartSLOduration=1.675999238 podStartE2EDuration="1.675999238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:31:42.673931233 +0000 UTC m=+3322.109885198" watchObservedRunningTime="2026-07-28 20:31:42.675999238 +0000 UTC m=+3322.111953230" Jul 28 20:31:42.763380 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763342 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763380 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763372 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnk2k\" (UniqueName: \"kubernetes.io/projected/88211e45-a9ad-44d9-b6e6-49366c91d7c8-kube-api-access-nnk2k\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763380 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763381 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-credential-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763392 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istiod-ca-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763402 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763410 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-envoy\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763418 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-data\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763425 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88211e45-a9ad-44d9-b6e6-49366c91d7c8-istio-podinfo\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.763632 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.763433 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88211e45-a9ad-44d9-b6e6-49366c91d7c8-workload-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:42.983304 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.983268 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:31:42.994034 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:42.994003 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e5feff-openshift-default-7dd4c777d6-j2dqx"] Jul 28 20:31:43.163831 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:43.163800 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" path="/var/lib/kubelet/pods/88211e45-a9ad-44d9-b6e6-49366c91d7c8/volumes" Jul 28 20:31:43.654577 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:43.654502 2562 generic.go:356] "Generic (PLEG): container finished" podID="b25743dd-31b2-46e6-8738-e2f16b560c4b" containerID="ef4a2f114dc04918c46ad6bd42e8f1c2c9726052a3c3eedaf2f40339a7d9a3d5" exitCode=0 Jul 28 20:31:43.655014 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:43.654576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" event={"ID":"b25743dd-31b2-46e6-8738-e2f16b560c4b","Type":"ContainerDied","Data":"ef4a2f114dc04918c46ad6bd42e8f1c2c9726052a3c3eedaf2f40339a7d9a3d5"} Jul 28 20:31:44.812211 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:44.812188 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:44.883593 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:44.883547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca\") pod \"b25743dd-31b2-46e6-8738-e2f16b560c4b\" (UID: \"b25743dd-31b2-46e6-8738-e2f16b560c4b\") " Jul 28 20:31:44.883950 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:44.883927 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b25743dd-31b2-46e6-8738-e2f16b560c4b" (UID: "b25743dd-31b2-46e6-8738-e2f16b560c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:31:44.985031 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:44.984945 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25743dd-31b2-46e6-8738-e2f16b560c4b-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:31:45.669108 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:45.669079 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" Jul 28 20:31:45.669295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:45.669113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb" event={"ID":"b25743dd-31b2-46e6-8738-e2f16b560c4b","Type":"ContainerDied","Data":"306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958"} Jul 28 20:31:45.669295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:45.669165 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306db9298f84eb332be09dda13e5f4ab5d6f3419f9c4a9eda45ba3e76c6d8958" Jul 28 20:31:57.195491 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:57.195448 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59"] Jul 28 20:31:57.201819 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:57.201792 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-aa68dd-29754510-ghw59"] Jul 28 20:31:57.479621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:57.479526 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb"] Jul 28 20:31:57.483156 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:57.483130 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aa68dd-vwlbb"] Jul 28 20:31:59.163519 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:59.163482 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25743dd-31b2-46e6-8738-e2f16b560c4b" path="/var/lib/kubelet/pods/b25743dd-31b2-46e6-8738-e2f16b560c4b/volumes" Jul 28 20:31:59.163962 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:31:59.163940 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be577848-eb89-449f-a107-c1dc3a700b6b" path="/var/lib/kubelet/pods/be577848-eb89-449f-a107-c1dc3a700b6b/volumes" Jul 28 20:32:42.418742 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.418696 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h"] Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419152 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419165 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419173 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419179 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419195 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25743dd-31b2-46e6-8738-e2f16b560c4b" containerName="revoke-keys" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419200 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25743dd-31b2-46e6-8738-e2f16b560c4b" containerName="revoke-keys" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419273 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:32:42.419379 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.419283 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25743dd-31b2-46e6-8738-e2f16b560c4b" containerName="revoke-keys" Jul 28 20:32:42.422578 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.422556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:42.425503 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.425465 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:32:42.429749 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.429723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h"] Jul 28 20:32:42.537809 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.537750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h\" (UID: \"911ab08a-fa97-4451-842a-505619ac99d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:42.639031 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.638990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h\" (UID: \"911ab08a-fa97-4451-842a-505619ac99d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:42.639697 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.639639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h\" (UID: \"911ab08a-fa97-4451-842a-505619ac99d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:42.737818 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:42.737722 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:43.094649 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:43.094618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h"] Jul 28 20:32:43.095892 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:32:43.095867 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911ab08a_fa97_4451_842a_505619ac99d2.slice/crio-03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f WatchSource:0}: Error finding container 03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f: Status 404 returned error can't find the container with id 03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f Jul 28 20:32:44.042742 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:44.042708 2562 generic.go:356] "Generic (PLEG): container finished" podID="911ab08a-fa97-4451-842a-505619ac99d2" containerID="ce69bfbc350529261dd5adda6aea33e7822f9721086c06db6a4a5c498fb8e2ce" exitCode=0 Jul 28 20:32:44.043135 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:44.042788 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" event={"ID":"911ab08a-fa97-4451-842a-505619ac99d2","Type":"ContainerDied","Data":"ce69bfbc350529261dd5adda6aea33e7822f9721086c06db6a4a5c498fb8e2ce"} Jul 28 20:32:44.043135 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:44.042830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" event={"ID":"911ab08a-fa97-4451-842a-505619ac99d2","Type":"ContainerStarted","Data":"03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f"} Jul 28 20:32:45.196940 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:45.196916 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:45.366003 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:45.365902 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca\") pod \"911ab08a-fa97-4451-842a-505619ac99d2\" (UID: \"911ab08a-fa97-4451-842a-505619ac99d2\") " Jul 28 20:32:45.366269 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:45.366248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "911ab08a-fa97-4451-842a-505619ac99d2" (UID: "911ab08a-fa97-4451-842a-505619ac99d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:32:45.467482 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:45.467445 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ab08a-fa97-4451-842a-505619ac99d2-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:32:46.057719 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:46.057688 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" Jul 28 20:32:46.057897 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:46.057721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h" event={"ID":"911ab08a-fa97-4451-842a-505619ac99d2","Type":"ContainerDied","Data":"03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f"} Jul 28 20:32:46.057897 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:46.057774 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03680cb86735bc28d4d43459256e93d747ca1e8eefcead0af33243d59568515f" Jul 28 20:32:52.983785 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:52.983742 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w"] Jul 28 20:32:52.987955 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:52.987928 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4ec840-29754510-j5s7w"] Jul 28 20:32:53.165116 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:53.165071 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed8198f-5194-4d03-b87a-6d0df004070d" path="/var/lib/kubelet/pods/6ed8198f-5194-4d03-b87a-6d0df004070d/volumes" Jul 28 20:32:53.244442 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:53.244354 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h"] Jul 28 20:32:53.248874 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:53.248841 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ec840-nzj4h"] Jul 28 20:32:55.163553 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:32:55.163523 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911ab08a-fa97-4451-842a-505619ac99d2" path="/var/lib/kubelet/pods/911ab08a-fa97-4451-842a-505619ac99d2/volumes" Jul 28 20:33:00.454736 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.454695 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd"] Jul 28 20:33:00.455233 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.455168 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911ab08a-fa97-4451-842a-505619ac99d2" containerName="revoke-keys" Jul 28 20:33:00.455233 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.455185 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="911ab08a-fa97-4451-842a-505619ac99d2" containerName="revoke-keys" Jul 28 20:33:00.455336 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.455277 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88211e45-a9ad-44d9-b6e6-49366c91d7c8" containerName="istio-proxy" Jul 28 20:33:00.455336 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.455291 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="911ab08a-fa97-4451-842a-505619ac99d2" containerName="revoke-keys" Jul 28 20:33:00.459776 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.459757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:00.462857 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.462836 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:33:00.468914 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.468887 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd"] Jul 28 20:33:00.494243 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.494197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd\" (UID: \"f19c3793-df83-4830-a0f4-7081f2c88d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:00.595330 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.595289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd\" (UID: \"f19c3793-df83-4830-a0f4-7081f2c88d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:00.596125 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.596099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd\" (UID: \"f19c3793-df83-4830-a0f4-7081f2c88d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:00.772226 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.772140 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:00.917101 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:00.917077 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd"] Jul 28 20:33:00.918345 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:33:00.918320 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19c3793_df83_4830_a0f4_7081f2c88d73.slice/crio-9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea WatchSource:0}: Error finding container 9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea: Status 404 returned error can't find the container with id 9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea Jul 28 20:33:01.164792 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:01.164760 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" event={"ID":"f19c3793-df83-4830-a0f4-7081f2c88d73","Type":"ContainerStarted","Data":"9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea"} Jul 28 20:33:02.168112 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:02.168077 2562 generic.go:356] "Generic (PLEG): container finished" podID="f19c3793-df83-4830-a0f4-7081f2c88d73" containerID="4e8fadd2f49c3354bc5a0b184defa80c7e78a0e986ac4bc0a9aa9495edfe6918" exitCode=0 Jul 28 20:33:02.168517 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:02.168165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" event={"ID":"f19c3793-df83-4830-a0f4-7081f2c88d73","Type":"ContainerDied","Data":"4e8fadd2f49c3354bc5a0b184defa80c7e78a0e986ac4bc0a9aa9495edfe6918"} Jul 28 20:33:03.317812 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:03.317785 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:03.421392 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:03.421363 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca\") pod \"f19c3793-df83-4830-a0f4-7081f2c88d73\" (UID: \"f19c3793-df83-4830-a0f4-7081f2c88d73\") " Jul 28 20:33:03.421766 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:03.421743 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f19c3793-df83-4830-a0f4-7081f2c88d73" (UID: "f19c3793-df83-4830-a0f4-7081f2c88d73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:33:03.522762 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:03.522638 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19c3793-df83-4830-a0f4-7081f2c88d73-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:04.181933 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:04.181903 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" Jul 28 20:33:04.182112 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:04.181938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd" event={"ID":"f19c3793-df83-4830-a0f4-7081f2c88d73","Type":"ContainerDied","Data":"9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea"} Jul 28 20:33:04.182112 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:04.181981 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ede547cd01751d41a6524a1fe86fad74c8573d902b5f21c20b6b969ea5b8aea" Jul 28 20:33:11.165188 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:11.165155 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7"] Jul 28 20:33:11.167315 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:11.167282 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a1e1ef-29754510-vltr7"] Jul 28 20:33:11.514454 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:11.514360 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd"] Jul 28 20:33:11.528671 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:11.528623 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a1e1ef-tbgqd"] Jul 28 20:33:13.163468 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:13.163427 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623e09fd-6429-4c19-be7f-ebd5f86119ea" path="/var/lib/kubelet/pods/623e09fd-6429-4c19-be7f-ebd5f86119ea/volumes" Jul 28 20:33:13.164077 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:13.164056 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19c3793-df83-4830-a0f4-7081f2c88d73" path="/var/lib/kubelet/pods/f19c3793-df83-4830-a0f4-7081f2c88d73/volumes" Jul 28 20:33:18.654821 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.654777 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62"] Jul 28 20:33:18.655343 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.655325 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19c3793-df83-4830-a0f4-7081f2c88d73" containerName="revoke-keys" Jul 28 20:33:18.655391 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.655345 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19c3793-df83-4830-a0f4-7081f2c88d73" containerName="revoke-keys" Jul 28 20:33:18.655428 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.655406 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19c3793-df83-4830-a0f4-7081f2c88d73" containerName="revoke-keys" Jul 28 20:33:18.658907 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.658889 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:18.662341 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.662315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7j5xk\"" Jul 28 20:33:18.666111 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.666085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62"] Jul 28 20:33:18.762516 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.762471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62\" (UID: \"250d8f5f-6ffd-41ac-b55a-80a7482cdc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:18.863332 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.863292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62\" (UID: \"250d8f5f-6ffd-41ac-b55a-80a7482cdc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:18.863968 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.863950 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62\" (UID: \"250d8f5f-6ffd-41ac-b55a-80a7482cdc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:18.971647 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:18.971528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:19.330495 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:19.330468 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62"] Jul 28 20:33:19.332760 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:33:19.332724 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250d8f5f_6ffd_41ac_b55a_80a7482cdc06.slice/crio-1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d WatchSource:0}: Error finding container 1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d: Status 404 returned error can't find the container with id 1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d Jul 28 20:33:20.298926 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:20.298889 2562 generic.go:356] "Generic (PLEG): container finished" podID="250d8f5f-6ffd-41ac-b55a-80a7482cdc06" containerID="52b3fa569ab8b2c6e13271431ee4243fef0052179cf52a03a177a7f61ad0863b" exitCode=0 Jul 28 20:33:20.299298 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:20.298980 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" event={"ID":"250d8f5f-6ffd-41ac-b55a-80a7482cdc06","Type":"ContainerDied","Data":"52b3fa569ab8b2c6e13271431ee4243fef0052179cf52a03a177a7f61ad0863b"} Jul 28 20:33:20.299298 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:20.299038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" event={"ID":"250d8f5f-6ffd-41ac-b55a-80a7482cdc06","Type":"ContainerStarted","Data":"1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d"} Jul 28 20:33:21.450103 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:21.450079 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:21.588976 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:21.588875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca\") pod \"250d8f5f-6ffd-41ac-b55a-80a7482cdc06\" (UID: \"250d8f5f-6ffd-41ac-b55a-80a7482cdc06\") " Jul 28 20:33:21.589313 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:21.589281 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "250d8f5f-6ffd-41ac-b55a-80a7482cdc06" (UID: "250d8f5f-6ffd-41ac-b55a-80a7482cdc06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:33:21.690411 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:21.690378 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250d8f5f-6ffd-41ac-b55a-80a7482cdc06-maas-api-service-ca\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:22.311804 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:22.311772 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" Jul 28 20:33:22.311989 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:22.311809 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62" event={"ID":"250d8f5f-6ffd-41ac-b55a-80a7482cdc06","Type":"ContainerDied","Data":"1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d"} Jul 28 20:33:22.311989 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:22.311859 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1005723598d6890acf45b5b64e6330f769a39b615ecdede25492dd27a91c591d" Jul 28 20:33:29.213710 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:29.213655 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9"] Jul 28 20:33:29.217242 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:29.217211 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-47f87c-29754510-m65t9"] Jul 28 20:33:29.395030 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:29.394998 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62"] Jul 28 20:33:29.398520 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:29.398496 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-47f87c-d7d62"] Jul 28 20:33:31.162952 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:31.162920 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250d8f5f-6ffd-41ac-b55a-80a7482cdc06" path="/var/lib/kubelet/pods/250d8f5f-6ffd-41ac-b55a-80a7482cdc06/volumes" Jul 28 20:33:31.163348 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:31.163205 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf4f444-18eb-4b6d-bab9-75fe4b3d127f" path="/var/lib/kubelet/pods/3bf4f444-18eb-4b6d-bab9-75fe4b3d127f/volumes" Jul 28 20:33:36.280885 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:36.280856 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:33:36.281361 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:36.281063 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" containerID="cri-o://5dbeb6740b22b11616d4767777972bdd19228a066575ce886fdb1ee7cff9edf9" gracePeriod=30 Jul 28 20:33:41.233225 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.233183 2562 patch_prober.go:28] interesting pod/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 20:33:41.233609 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.233257 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 20:33:41.438981 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.438946 2562 generic.go:356] "Generic (PLEG): container finished" podID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerID="5dbeb6740b22b11616d4767777972bdd19228a066575ce886fdb1ee7cff9edf9" exitCode=0 Jul 28 20:33:41.438981 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.438971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerDied","Data":"5dbeb6740b22b11616d4767777972bdd19228a066575ce886fdb1ee7cff9edf9"} Jul 28 20:33:41.439219 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.439044 2562 scope.go:117] "RemoveContainer" containerID="e708eafc8c1f7f93de0a40c29844851032536139bd59be50e79b389ef733b828" Jul 28 20:33:41.540709 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.540683 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:33:41.581133 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581095 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581133 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581130 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581151 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581187 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581227 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581289 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g8tj\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581327 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581383 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581741 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581393 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs\") pod \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\" (UID: \"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d\") " Jul 28 20:33:41.581938 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581858 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:33:41.582088 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.581970 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:33:41.582088 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.582059 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:33:41.582216 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.582154 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:33:41.583335 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.583299 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:33:41.583634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.583610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:33:41.583916 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.583887 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:33:41.584023 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.583960 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj" (OuterVolumeSpecName: "kube-api-access-6g8tj") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "kube-api-access-6g8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:33:41.584089 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.584033 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" (UID: "a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:33:41.682676 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682638 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8tj\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-kube-api-access-6g8tj\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682691 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-credential-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682713 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istiod-ca-cert\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682723 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-certs\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682732 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-envoy\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682740 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-workload-socket\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682748 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-podinfo\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682755 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-token\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:41.682860 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:41.682763 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d-istio-data\") on node \"ip-10-0-132-248.ec2.internal\" DevicePath \"\"" Jul 28 20:33:42.277054 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.277016 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dw87p_5bef0052-9412-40b4-a364-01ed66d9ac6f/manager/0.log" Jul 28 20:33:42.407026 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.406992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qkhvq_69d9ce3b-6cd9-4051-83b6-cb5c92da7d05/manager/0.log" Jul 28 20:33:42.446334 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.446302 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" Jul 28 20:33:42.446534 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.446340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp" event={"ID":"a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d","Type":"ContainerDied","Data":"64486442395cfed1209a5c3b6482de6555112dcfdd56bf2cb16683d736594e3d"} Jul 28 20:33:42.446534 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.446398 2562 scope.go:117] "RemoveContainer" containerID="5dbeb6740b22b11616d4767777972bdd19228a066575ce886fdb1ee7cff9edf9" Jul 28 20:33:42.474278 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.474252 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:33:42.478819 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.478794 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-47f87c-openshift-default-7d98b58d64-twgpp"] Jul 28 20:33:42.531473 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.531383 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c5fbf75cf-sttpr_3e097f78-ace7-40d9-8c02-1588c0374336/manager/0.log" Jul 28 20:33:42.648621 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.648593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5ktdt_e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7/server/0.log" Jul 28 20:33:42.890727 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.890646 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-cpjrr_2d55e6cb-7b24-4081-ab58-842287f4e250/copy-manifests/0.log" Jul 28 20:33:42.909358 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:42.909331 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-cpjrr_2d55e6cb-7b24-4081-ab58-842287f4e250/manager/0.log" Jul 28 20:33:43.163616 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:43.163579 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" path="/var/lib/kubelet/pods/a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d/volumes" Jul 28 20:33:43.547229 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:43.547141 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7968df7757-dn8cc_5b7a9f43-1f5f-42d5-95f3-167d4d0e2515/maas-api/0.log" Jul 28 20:33:43.667226 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:43.667194 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754480-nxz29_eea00239-b3b8-4127-bc9a-37a95dbd1590/cleanup/0.log" Jul 28 20:33:43.777737 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:43.777714 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754495-4tcgn_9a9cba81-5fa8-4367-a924-677854c89356/cleanup/0.log" Jul 28 20:33:43.885271 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:43.885199 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754510-cslgj_b41d5476-aba9-42df-ac10-351c8f908af2/cleanup/0.log" Jul 28 20:33:44.003295 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.003266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8rvkn_1702c138-b75c-4ba4-9d65-04c9ae73d5b3/postgres/0.log" Jul 28 20:33:44.799578 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.799554 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/util/0.log" Jul 28 20:33:44.806351 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.806331 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/pull/0.log" Jul 28 20:33:44.812076 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.812051 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/extract/0.log" Jul 28 20:33:44.921796 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.921768 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/util/0.log" Jul 28 20:33:44.928109 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.928087 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/pull/0.log" Jul 28 20:33:44.934395 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:44.934375 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/extract/0.log" Jul 28 20:33:45.051181 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.051089 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/util/0.log" Jul 28 20:33:45.057538 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.057516 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/pull/0.log" Jul 28 20:33:45.063372 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.063353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/extract/0.log" Jul 28 20:33:45.170713 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.170684 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/pull/0.log" Jul 28 20:33:45.182781 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.182756 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/extract/0.log" Jul 28 20:33:45.188886 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.188862 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/util/0.log" Jul 28 20:33:45.427031 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.427006 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-prsdm_d4e17283-4328-4fbf-a98b-ac3fe8779270/manager/0.log" Jul 28 20:33:45.536882 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.536855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ltg8m_ead17357-921b-4f93-9593-d7f44df3e381/manager/0.log" Jul 28 20:33:45.655600 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.655575 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lxlmj_d096c60c-e8e3-427f-aeb9-a15d1036f0da/kuadrant-console-plugin/0.log" Jul 28 20:33:45.776302 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.776216 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-62ld4_66f98abf-d517-4cf7-a99f-3e811610ed9c/registry-server/0.log" Jul 28 20:33:45.893209 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:45.893180 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qtt5s_5526c3d9-8963-4ed8-85eb-c8012ed42cfe/manager/0.log" Jul 28 20:33:46.865608 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:46.865578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-vsl7h_2811d12a-4ad1-45ff-87f7-ea72a4225309/kube-auth-proxy/0.log" Jul 28 20:33:46.995045 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:46.995017 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/1.log" Jul 28 20:33:47.123022 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:47.122941 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vc598_b38c912a-eba9-4393-a45d-04ae41664576/payload-pre-processing/0.log" Jul 28 20:33:47.262522 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:47.262491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:33:47.371203 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:47.371173 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bdcf959c-db97m_630164a9-08ce-432d-ba5a-f2047a822a82/router/0.log" Jul 28 20:33:48.047127 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:48.047098 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5s4kt_a16044ea-b5f3-4cf6-87f4-8ac8c2929bc7/main/0.log" Jul 28 20:33:48.164529 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:48.164502 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ppbd2_374aa924-34bb-428f-85d5-5e3872714f67/main/0.log" Jul 28 20:33:53.985131 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985097 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnmjx/must-gather-xwvvc"] Jul 28 20:33:53.985744 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985725 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.985841 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985746 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.985841 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985768 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250d8f5f-6ffd-41ac-b55a-80a7482cdc06" containerName="revoke-keys" Jul 28 20:33:53.985841 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985776 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="250d8f5f-6ffd-41ac-b55a-80a7482cdc06" containerName="revoke-keys" Jul 28 20:33:53.986011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985897 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.986011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.985913 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="250d8f5f-6ffd-41ac-b55a-80a7482cdc06" containerName="revoke-keys" Jul 28 20:33:53.986011 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.986003 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.986174 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.986014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.986174 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.986134 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17b36cf-8bc4-4b2f-aa27-6b0cf337e72d" containerName="istio-proxy" Jul 28 20:33:53.991003 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.990980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:53.993730 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.993707 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bnmjx\"/\"default-dockercfg-8nlqz\"" Jul 28 20:33:53.993939 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.993925 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnmjx\"/\"kube-root-ca.crt\"" Jul 28 20:33:53.993995 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.993960 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnmjx\"/\"openshift-service-ca.crt\"" Jul 28 20:33:53.998978 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:53.998430 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/must-gather-xwvvc"] Jul 28 20:33:54.098933 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.098901 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15230a02-9538-47f7-a30e-760d32d77843-must-gather-output\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.099097 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.098967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxz5g\" (UniqueName: \"kubernetes.io/projected/15230a02-9538-47f7-a30e-760d32d77843-kube-api-access-jxz5g\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.200197 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.200166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxz5g\" (UniqueName: \"kubernetes.io/projected/15230a02-9538-47f7-a30e-760d32d77843-kube-api-access-jxz5g\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.200420 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.200234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15230a02-9538-47f7-a30e-760d32d77843-must-gather-output\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.200709 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.200655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15230a02-9538-47f7-a30e-760d32d77843-must-gather-output\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.210766 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.210740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxz5g\" (UniqueName: \"kubernetes.io/projected/15230a02-9538-47f7-a30e-760d32d77843-kube-api-access-jxz5g\") pod \"must-gather-xwvvc\" (UID: \"15230a02-9538-47f7-a30e-760d32d77843\") " pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.301397 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.301305 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" Jul 28 20:33:54.455719 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.455677 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/must-gather-xwvvc"] Jul 28 20:33:54.458235 ip-10-0-132-248 kubenswrapper[2562]: W0728 20:33:54.458208 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15230a02_9538_47f7_a30e_760d32d77843.slice/crio-284afb29441b5055910c5b2ec190ca2a7d79d649728d726f6f56b6ab7f952033 WatchSource:0}: Error finding container 284afb29441b5055910c5b2ec190ca2a7d79d649728d726f6f56b6ab7f952033: Status 404 returned error can't find the container with id 284afb29441b5055910c5b2ec190ca2a7d79d649728d726f6f56b6ab7f952033 Jul 28 20:33:54.460177 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.460162 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:33:54.529043 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:54.529012 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" event={"ID":"15230a02-9538-47f7-a30e-760d32d77843","Type":"ContainerStarted","Data":"284afb29441b5055910c5b2ec190ca2a7d79d649728d726f6f56b6ab7f952033"} Jul 28 20:33:55.540596 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:55.540565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" event={"ID":"15230a02-9538-47f7-a30e-760d32d77843","Type":"ContainerStarted","Data":"588b405965f038ebe426a028cac0c8a084f48a946a5a80dc8aed968cbbdfc145"} Jul 28 20:33:55.540962 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:55.540607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" event={"ID":"15230a02-9538-47f7-a30e-760d32d77843","Type":"ContainerStarted","Data":"257ba59d5d292b2d61125a7c2ab82d6b668c74afda061417c407112fd0918301"} Jul 28 20:33:55.559345 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:55.559295 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnmjx/must-gather-xwvvc" podStartSLOduration=1.6843275439999998 podStartE2EDuration="2.559278985s" totalImagesPullingTime="874.951441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:33:55.55716667 +0000 UTC m=+3454.993120624" watchObservedRunningTime="2026-07-28 20:33:55.559278985 +0000 UTC m=+3454.995232936" Jul 28 20:33:57.017443 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:57.017407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kqqfs_76938137-378f-4f48-99ab-dae7a5688d58/global-pull-secret-syncer/0.log" Jul 28 20:33:57.121175 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:57.121143 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-82zr4_8d1e8efb-4f43-486e-9192-11fd9a39c941/konnectivity-agent/0.log" Jul 28 20:33:57.192359 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:33:57.192324 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-248.ec2.internal_65b35075ce250b6dbc5ab26eb37c0081/haproxy/0.log" Jul 28 20:34:00.199657 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.199615 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/extract/0.log" Jul 28 20:34:00.283218 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.283183 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/util/0.log" Jul 28 20:34:00.352722 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.352694 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mktmj_271d3d76-5efa-4337-9fe9-def48a6ee259/pull/0.log" Jul 28 20:34:00.448494 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.448461 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/extract/0.log" Jul 28 20:34:00.502934 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.502855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/util/0.log" Jul 28 20:34:00.560516 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.560488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07cbdz_f36cc7cf-d61a-4270-9199-baeb90d78dbe/pull/0.log" Jul 28 20:34:00.608642 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.608609 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/extract/0.log" Jul 28 20:34:00.649764 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.649733 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/util/0.log" Jul 28 20:34:00.688864 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.688833 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xzrpw_53aad1db-8cad-44da-be15-9d65419096bb/pull/0.log" Jul 28 20:34:00.769817 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.769727 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/extract/0.log" Jul 28 20:34:00.819527 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.819319 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/util/0.log" Jul 28 20:34:00.870548 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:00.870517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1clxsr_415a9325-f24b-4704-9204-1ff2cb43d51d/pull/0.log" Jul 28 20:34:01.294427 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:01.294388 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-prsdm_d4e17283-4328-4fbf-a98b-ac3fe8779270/manager/0.log" Jul 28 20:34:01.344491 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:01.344457 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ltg8m_ead17357-921b-4f93-9593-d7f44df3e381/manager/0.log" Jul 28 20:34:01.396204 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:01.396178 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lxlmj_d096c60c-e8e3-427f-aeb9-a15d1036f0da/kuadrant-console-plugin/0.log" Jul 28 20:34:01.464738 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:01.464515 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-62ld4_66f98abf-d517-4cf7-a99f-3e811610ed9c/registry-server/0.log" Jul 28 20:34:01.683275 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:01.683240 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qtt5s_5526c3d9-8963-4ed8-85eb-c8012ed42cfe/manager/0.log" Jul 28 20:34:03.735832 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:03.735796 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-tbm99_89664ed0-d7e1-4548-bf7c-d923c48b3ede/cluster-monitoring-operator/0.log" Jul 28 20:34:04.187433 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.187386 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8xzf_67cd925f-253f-4df9-b7f0-53a645bf0386/node-exporter/0.log" Jul 28 20:34:04.219501 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.219464 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8xzf_67cd925f-253f-4df9-b7f0-53a645bf0386/kube-rbac-proxy/0.log" Jul 28 20:34:04.263935 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.263869 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8xzf_67cd925f-253f-4df9-b7f0-53a645bf0386/init-textfile/0.log" Jul 28 20:34:04.671963 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.671919 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69f797dc9d-zsslk_2cf68961-85dc-4df6-af7e-9a81b397e9be/telemeter-client/0.log" Jul 28 20:34:04.699232 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.699149 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69f797dc9d-zsslk_2cf68961-85dc-4df6-af7e-9a81b397e9be/reload/0.log" Jul 28 20:34:04.726631 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:04.726601 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69f797dc9d-zsslk_2cf68961-85dc-4df6-af7e-9a81b397e9be/kube-rbac-proxy/0.log" Jul 28 20:34:06.823900 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:06.823871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/1.log" Jul 28 20:34:06.832445 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:06.832416 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-576sb_dce9f717-6cac-46ae-9efc-70444d11deea/console-operator/2.log" Jul 28 20:34:06.885089 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:06.885051 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq"] Jul 28 20:34:06.892041 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:06.892011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:06.903310 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:06.903283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq"] Jul 28 20:34:07.063396 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.063362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-proc\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.063608 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.063472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-sys\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.063608 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.063595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/f4246201-45ad-44ac-a8da-606b630bcaca-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.063755 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.063633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-podres\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.063755 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.063686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-lib-modules\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164329 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/f4246201-45ad-44ac-a8da-606b630bcaca-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164531 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-podres\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164531 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-lib-modules\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164531 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-proc\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164531 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-sys\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.164815 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.164607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-sys\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.165048 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.165022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-podres\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.165137 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.165121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-lib-modules\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.165197 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.165172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4246201-45ad-44ac-a8da-606b630bcaca-proc\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.175330 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.175293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/f4246201-45ad-44ac-a8da-606b630bcaca-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-mp7kq\" (UID: \"f4246201-45ad-44ac-a8da-606b630bcaca\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.207206 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.207165 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.433541 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.430725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq"] Jul 28 20:34:07.501033 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.500991 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-wwv56_9ed56166-4e0c-4475-b141-49a89fcd2a30/download-server/0.log" Jul 28 20:34:07.665434 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.665379 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" event={"ID":"f4246201-45ad-44ac-a8da-606b630bcaca","Type":"ContainerStarted","Data":"569e859450d375dbbfcb87ae486e39d67c176e1c990ba8bfeb15678bc28c4589"} Jul 28 20:34:07.665634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.665459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" event={"ID":"f4246201-45ad-44ac-a8da-606b630bcaca","Type":"ContainerStarted","Data":"70963a2e21de42fbd0306886d458441163a9f8dd984ed6cfdd3164bdac0ecfe1"} Jul 28 20:34:07.665634 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.665583 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:07.689938 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:07.689835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" podStartSLOduration=1.689817906 podStartE2EDuration="1.689817906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:07.68537327 +0000 UTC m=+3467.121327225" watchObservedRunningTime="2026-07-28 20:34:07.689817906 +0000 UTC m=+3467.125771861" Jul 28 20:34:08.241257 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:08.241229 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-hrvzb_9a2e98f5-0401-4e17-b78a-2a3fdd5eeed2/volume-data-source-validator/0.log" Jul 28 20:34:09.386545 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:09.386518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lk7lh_ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f/dns/0.log" Jul 28 20:34:09.410088 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:09.410061 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lk7lh_ff4cc05e-dbce-4ef2-a305-9a5a54f63e2f/kube-rbac-proxy/0.log" Jul 28 20:34:09.514728 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:09.514693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flg66_023e36af-fadf-47f4-93de-719edd61b816/dns-node-resolver/0.log" Jul 28 20:34:10.090348 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:10.090321 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84b668485b-szs7c_88d1cc11-3e0f-4717-af58-7d8adf7a8345/registry/0.log" Jul 28 20:34:10.123585 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:10.123556 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6g2s4_8e7a676f-2b30-4b7a-ba6b-606a21ca6068/node-ca/0.log" Jul 28 20:34:11.340624 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.340593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-vsl7h_2811d12a-4ad1-45ff-87f7-ea72a4225309/kube-auth-proxy/0.log" Jul 28 20:34:11.372537 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.372503 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/0.log" Jul 28 20:34:11.433781 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.433751 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/1.log" Jul 28 20:34:11.486974 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.486941 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vc598_b38c912a-eba9-4393-a45d-04ae41664576/payload-pre-processing/0.log" Jul 28 20:34:11.531872 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.531839 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:34:11.562497 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:11.562464 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bdcf959c-db97m_630164a9-08ce-432d-ba5a-f2047a822a82/router/0.log" Jul 28 20:34:12.386057 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:12.386027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-65trs_61168e6a-d37c-46a1-9b88-2e0e635c81b9/serve-healthcheck-canary/0.log" Jul 28 20:34:13.033992 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.033961 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dvczk_f4c57e53-4a0a-4b42-ade0-44ed7a500ed7/insights-operator/0.log" Jul 28 20:34:13.036029 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.036006 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dvczk_f4c57e53-4a0a-4b42-ade0-44ed7a500ed7/insights-operator/1.log" Jul 28 20:34:13.144925 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.144887 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkv6s_86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5/kube-rbac-proxy/0.log" Jul 28 20:34:13.168919 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.168892 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkv6s_86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5/exporter/0.log" Jul 28 20:34:13.196255 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.196217 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkv6s_86af7b69-99fe-4c8d-8c4b-3ed9cd4420f5/extractor/0.log" Jul 28 20:34:13.693179 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:13.693151 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-mp7kq" Jul 28 20:34:15.522932 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:15.522902 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dw87p_5bef0052-9412-40b4-a364-01ed66d9ac6f/manager/0.log" Jul 28 20:34:15.997436 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:15.997389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qkhvq_69d9ce3b-6cd9-4051-83b6-cb5c92da7d05/manager/0.log" Jul 28 20:34:16.272452 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:16.272372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c5fbf75cf-sttpr_3e097f78-ace7-40d9-8c02-1588c0374336/manager/0.log" Jul 28 20:34:16.310649 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:16.310617 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5ktdt_e2a1c34a-b4fb-4866-bf3b-c8f7f3d514a7/server/0.log" Jul 28 20:34:16.565293 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:16.564367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-cpjrr_2d55e6cb-7b24-4081-ab58-842287f4e250/manager/0.log" Jul 28 20:34:16.585726 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:16.585693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-cpjrr_2d55e6cb-7b24-4081-ab58-842287f4e250/copy-manifests/0.log" Jul 28 20:34:17.669862 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.669834 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-vsl7h_2811d12a-4ad1-45ff-87f7-ea72a4225309/kube-auth-proxy/0.log" Jul 28 20:34:17.696049 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.696001 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/0.log" Jul 28 20:34:17.756132 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.756067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/1.log" Jul 28 20:34:17.785310 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.785274 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vc598_b38c912a-eba9-4393-a45d-04ae41664576/payload-pre-processing/0.log" Jul 28 20:34:17.823450 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.823415 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:34:17.840181 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:17.840154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bdcf959c-db97m_630164a9-08ce-432d-ba5a-f2047a822a82/router/0.log" Jul 28 20:34:19.515118 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.515035 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-vsl7h_2811d12a-4ad1-45ff-87f7-ea72a4225309/kube-auth-proxy/0.log" Jul 28 20:34:19.543832 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.543790 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/0.log" Jul 28 20:34:19.599737 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.599691 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-f9dwd_2282a22c-3b4f-4477-a047-13cd7ebb7d6e/istio-proxy/1.log" Jul 28 20:34:19.633757 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.633724 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vc598_b38c912a-eba9-4393-a45d-04ae41664576/payload-pre-processing/0.log" Jul 28 20:34:19.667865 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.667827 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hkh7f_b8592493-106c-4024-b9df-9a8893a28016/payload-processing/0.log" Jul 28 20:34:19.689845 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:19.689815 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bdcf959c-db97m_630164a9-08ce-432d-ba5a-f2047a822a82/router/0.log" Jul 28 20:34:23.736681 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:23.736635 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g_45bcbaaa-d4fd-4c39-89fb-3c05b8102493/kube-storage-version-migrator-operator/1.log" Jul 28 20:34:23.738128 ip-10-0-132-248 kubenswrapper[2562]: I0728 20:34:23.738105 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-6lq5g_45bcbaaa-d4fd-4c39-89fb-3c05b8102493/kube-storage-version-migrator-operator/0.log"